1 / 22

VIBES Variational Inference Engine For Bayesian Networks

VIBES Variational Inference Engine For Bayesian Networks. John Winn. Inference Group, Cavendish Laboratory. in association with Chris Bishop (Microsoft Research, Cambridge) David Spiegelhalter (MRC Biostatistics Unit, Cambridge). 30 th October 2002. Overview. Bayesian Networks

maxima
Download Presentation

VIBES Variational Inference Engine For Bayesian Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VIBESVariational Inference Engine For Bayesian Networks John Winn Inference Group, Cavendish Laboratory in association with Chris Bishop (Microsoft Research, Cambridge) David Spiegelhalter (MRC Biostatistics Unit, Cambridge) 30th October 2002

  2. Overview • Bayesian Networks • Variational Inference • VIBES • Digit data demo

  3. P(b) P(f) Battery Fuel • Nodes represent variables P(g|b,f) • Links show dependencies Gauge P(t|b) • Conditional distributions at each node P(s|f,t) TurnsOver Starts • Network defines a joint distribution: P(b,f,g,t,s) = P(b)P(f)P(g|b,f)P(t|b)P(s|f,t) Bayesian Networks • Directed graph

  4. Battery Fuel Gauge Inference involves finding: TurnsOver P(H1, H2…| V) Starts Inference in Bayes Nets Observed variables V and hidden variables H. Hidden Observed (car doesn’t start)

  5. Approximate true posterior with: variational distribution Approximate Inference We want to find: typically can’t evaluate

  6. Variational Inference (in three easy steps…) • Choose Q(H|θ), a variational distribution with parameters θ. • Use KL divergence as a measure of ‘distance’ between P and Q. • Change parameters θ to minimiseKL(Q||P)

  7. KL Divergence For unimodal Q and bi-modal P: Q Minimising KL(Q||P) P Minimising KL(P||Q) Q P

  8. Minimising the KL divergence Start with the logevidence: This is L(Q) - a lower bound on ln P(V).

  9. KL(Q || P) ln P(V) L(Q) Minimising the KL divergence ‘distance’ evidence bound

  10. Battery Battery Fuel Fuel Gauge Gauge TurnsOver TurnsOver Starts P Q Factorised Qdistribution One possible choice of Q: i

  11. Maximising the bound For any factor, can maximise the bound in one step:

  12. xn   tn W N Example: Bayesian PCA Reduced data Component ‘selector’ Components Noise Data Plate “N copies of”

  13. PCA update equations Can we do this automatically ? from Bishop 1999

  14. e.g. Gaussian Exponential Family . . . Y1 Yk Choose conditionals P(X|Y) to be from the exponential family: X natural parameter vector u function

  15. conjugacy requires that: parentu function Conjugate-Exponential . . . Y1 Yk . . . cpj X . . . Z1 Zj

  16. myx mzjx mz1x Variational update for X Variational message passing Y1 Yk X . . . Z1 Zj

  17. mxβ mxμ Example: Gaussian mean (Gaussian) precision (Gamma) μ β x

  18. A function node f(A,B) is tractable if: Deterministic nodes Allow addition, multiplication, mixtures… P(X | A.B + C,Y) A B × C + Y X

  19. VIBES Variational Inference in Bayesian networks • Assumes factorised Q • Local updates by message passing • Works with any conjugate-exponential model • Just draw your model, specify distributions and press go!

  20. Bayesian PCA (again) xn  W  tn N

  21. Digits demo

  22. Work in progress • Allowing structure in the Q distribution (no longer fully factorised) • First release version of VIBES

More Related