CSAIL Publications and Digital Archive header
bullet Technical Reports bullet Work Products bullet Research Abstracts bullet Historical Collections bullet

link to publications.csail.mit.edu link to www.csail.mit.edu horizontal line


Research Abstracts - 2006
horizontal line

horizontal line

vertical line
vertical line

Parameter Expanded Variational Bayesian Methods

Yuan (Alan) Qi & Tommi S. Jaakkola


Bayesian inference has become increasingly important in statistical machine learning. Exact Bayesian calculations are often not feasible in practice, however. A number of approximate Bayesian methods have been proposed to make such calculations practical, among them the variational Bayesian (VB) approach. The VB approach, while useful, can nevertheless suffer from slow convergence to the approximate solution. To address this problem, we propose Parameter-eXpanded Variational Bayesian (PX-VB) methods to speed up VB. The new algorithm is inspired by parameter-expanded expectation maximization (PX-EM) and parameter-expanded data augmentation (PX-DA). PX-EM and -DA speed up EM and DA sampling by employing expanded auxiliary variables in the model. We analyze the convergence rates of VB and PX-VB and demonstrate the superior convergence rates of PX-VB in variational probit regression and automatic relevance determination.

  • Experimental results

    [1] M.J. Beal. Variational Algorithms for Approximate Bayesian Inference. PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.

    [2] C.M. Bishop and M. E. Tipping. Variational relevance vector machines. In 16th UAI, 2000.

    [3] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods in graphical models. In M. I. Jordan, editor, Learning in Graphical Models, 1998.

    [4] C. Liu, D. B. Rubin, and Y. N. Wu. Parameter expansion to accelerate EM: the PX-EM algorithm. Biometrika, 85:755–770, 1998.

    [5] Z. Q. Luo and P. Tseng. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72(1):7–35, Jan. 1992.

    [6] D. J. MacKay. Bayesian interpolation. Neural Computation, 4(3):415–447, 1992.

    [7] M. E. Tipping. The relevance vector machine. In NIPS, volume 12, pages 652–658. The MIT Press, 2000.

    [8] D. A. van Dyk and X. L. Meng. The art of data augmentation (with discussion). Journal of Computational and Graphical Statistics, 10(1):1–111, March 2001.


vertical line
vertical line
horizontal line

MIT logo Computer Science and Artificial Intelligence Laboratory (CSAIL)
The Stata Center, Building 32 - 32 Vassar Street - Cambridge, MA 02139 - USA
tel:+1-617-253-0073 - publications@csail.mit.edu