1 / 33

Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models

Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models. Mohammad Emti yaz Khan Joint work with Benjamin Marlin, and Kevin Murphy University of British Columbia June 29, 2011. Modeling Binary Data. Main Topic of our paper

cosmo
Download Presentation

Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models Mohammad Emtiyaz Khan Joint work with Benjamin Marlin,and Kevin Murphy University of British Columbia June 29, 2011

  2. Modeling Binary Data Main Topic of our paper Bernoulli-Logistic Latent Gaussian Models (bLGMs) Image from http://thenextweb.com/in/2011/06/06/india-to-join-the-open-data-revolution-in-july/

  3. bLGMs - Classification Models Bayesian Logistic Regression and Gaussian Process Classification (Jaakkola and Jordan 1996, Rasmussen 2004, Gibbs and Mackay 2000, Kuss and Rasmussen 2006, Nickisch and Rasmussen 2008, Kim and Ghahramani, 2003). Figures reproduced using GPML toolbox

  4. bLGMs - Latent Factor Models Probabilistic PCA and Factor Analysis models (Tipping 1999, Collins, Dasgupta and Schapire 2001, Mohammed, Heller, and Ghahramani 2008, Girolami 2001, Yu and Tresp 2004).

  5. x Parameter Learning is Intractable Logistic Likelihood is not conjugate to the Gaussian prior. We propose piecewise bounds to obtain tractable lower bounds to marginal likelihood.

  6. Learning in bLGMs

  7. Bernoulli-Logistic Latent Gaussian Models µ Σ zLn z1n z2n Parameter Set yDn y3n y2n y1n n=1:N W

  8. Learning Parameters of bLGMs x

  9. Variational Lower Bound (Jensen’s) some other tractable terms in m and V + x

  10. Quadratic Bounds x Bohning’s bound (Bohning, 1992)

  11. Quadratic Bounds x Bohning’s bound (Bohning, 1992) Jaakkola’s bound (Jaakkola and Jordan, 1996) Both bounds have unbounded error.

  12. Problems with Quadratic Bounds µ σ 1-D example with µ = 2, σ = 2 zn Generate data, fix µ = 2, and compare marginal likelihood and lower bound wrt σ As this is a 1-D problem, we can compute lower bounds without Jensen’s inequality. So plots that follow have errors only due to error in bounds. yn n=1:N

  13. Problems with Quadratic Bounds Bohning Jaakkola Piecewise Q1(x) Q2(x) Q3(x)

  14. Piecewise Bounds

  15. Finding Piecewise Bounds Find Cut points, and parameters of each pieces by minimizing maximum error. Linear pieces (Hsiung, Kim and Boyd, 2008) Quadratic Pieces (Nelder-Mead method) Fixed Piecewise Bounds! Increase accuracy by increasing number of pieces.

  16. Linear Vs Quadratic

  17. Results

  18. Binary Factor Analysis (bFA) UCI voting dataset with D=15, N=435. Train-test split 80-20% Compare cross-entropy error on missing value prediction on test data. ZLn Z1n Z2n YDn Y3n Y2n Y1n n=1:N W

  19. bFA – Error vs Time Bohning Jaakkola Piecewise Linear with 3 pieces Piecewise Quad with 3 pieces Piecewise Quad with 10 pieces

  20. bFA – Error Across Splits Bohning Jaakkola Error with Piecewise Quadratic Error with Bohning and Jaakkola

  21. Gaussian Process Classification X σ s • We repeat the experiments described in Kuss and Rasmussen, 2006 • We set µ =0 and squared exponential Kernel Σij=σ exp[(xi-xj)^2/s] • Estimateσ and s. • We run experiments on Ionoshphere (D = 200) • Compare Cross-entropy Prediction Error for test data. µ Σ zD z1 z2 y1 y2 yD

  22. GP – Marginal Likelihood

  23. GP – Prediction Error

  24. EP vs Variational We see that the variational approach underestimates the marginal likelihood in some regions of parameter space. However, both methods give comparable results for prediction error. In general, the variational EM algorithm for parameter learning is guaranteed to converge when appropriate numerical methods are used, Nickisch and Rasmussen (2008) describe the variational approach as more principled than EP.

  25. Conclusions

  26. Conclusions Fixed piecewise bounds can give a significant improvement in estimation and prediction accuracy relative to variational quadratic bounds. We can drive the error in the logistic-log-partition bound to zero by letting the number of pieces increase. This increase in accuracy comes with a corresponding increase in computation time. Unlike many other frameworks, we have a very fine grained control over the speed-accuracy trade-off through controlling the number of pieces in the bound.

  27. Thank You

  28. Piecewise-Bounds: Optimization Problem

  29. Latent Gaussian Graphical Model LED dataset, 24 variables, N=2000 µ Σ zDn z1n z2n y1n y3n yDn n=1:N

  30. Sparse Version

More Related