Approximate Bayesian Learning Using Bregman Divergence

1. Approximate Bayesian Learning Using Bregman Divergence Kazuho Watanabe Nara Institute of Science and Technology

2. Outline • General framework of LVA using Bregman divergence • Its application to latent variable models Local Variational Approximation (LVA) for Bayesian Learning logistic regression (Jaakkola & Jordan, 2000) sparse linear model (Seeger, 2008) Contents:

3. Bayesian Learning Training data: Parameter: Model: Prior: Posterior： intractable Marginal likelihood： Predictive distribution:

4. Example: Logistic Regression Likelihood: Prior: intractable

5. Local Variational Approximation Normalize Variational parameter Approximating Posteriors

6. Local Variational Approximation (Example) (Jaakkola & Jordan, 2000) Convex w.r.t. quadratic function of w : normal Convex w.r.t. : normal linear function of w

7. General Framework of LVA based on Convexity ,　　　　：convex functions Bregman divergence Example:

8. Free Energy Bounds : Free energy Upper bound minimization :posterior :approximation EMalgorithm: Lower bound maximization :posterior :approximation

9. Latent Variable Models Latent variables: Model: Ex) Gaussian mixture model, Hidden Markov model

10. LVA for Latent Variable Models Variational free energy Approximating posterior Variational Bayes

11. Asymptotic Analysis of Variational Free Energy :Entropy i.i.d. Variational Bayes cf.) Bayes (S.Watanabe, 2001) : log-sum inequality

12. Conclusion General framework for local variational approximation using Bregman divergence Its application to latent variable models