1 / 63

Expectation Propagation for Graphical Models

Expectation Propagation for Graphical Models. Yuan (Alan) Qi Joint work with Tom Minka. Motivation. Graphical models are widely used in real-world applications, such as wireless communications and bioinformatics.

jack-park
Download Presentation

Expectation Propagation for Graphical Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Expectation Propagation for Graphical Models Yuan (Alan) Qi Joint work with Tom Minka

  2. Motivation • Graphical models are widely used in real-world applications, such as wireless communications and bioinformatics. • Inference techniques on graphical models often sacrifice efficiency for accuracy or sacrifice accuracy for efficiency. • Need a new method that better balances the trade-off between accuracy and efficiency.

  3. What we want Motivation Accuracy Current Techniques Efficiency

  4. Outline • Background • Expectation Propagation (EP) on dynamic systems • Poisson tracking • Signal detection for wireless communications • Tree-structured EP on loopy graphs • Conclusions and future work

  5. Outline • Background • Expectation Propagation (EP) on dynamic systems • Poisson tracking • Signal detection for wireless communications • Tree-structured EP on loopy graphs • Conclusions

  6. x1 x2 x1 x2 y1 y2 y1 y2 x1 x2 x1 x2 y1 y2 y1 y2 Graphical Models

  7. Inference on Graphical Models • Bayesian inference techniques: • Belief propagation(BP): Kalman filtering /smoothing, forward-backward algorithm • Monte Carlo: Particle filter/smoothers, MCMC • Loopy BP: typically efficient, but not accurate • Monte Carlo: accurate, but often not efficient

  8. Efficiency vs. Accuracy MC EP ? Accuracy BP Efficiency

  9. Expectation Propagation in a Nutshell • Approximate a probability distribution by simpler parametric terms: • Each approximation term lives in an exponential family (e.g. Gaussian)

  10. Update Term Approximation • Iterate the fixed-point equation by moment matching: Where the leave-one-out approximation is

  11. Outline • Background • Expectation Propagation (EP) on dynamic systems • Poisson tracking • Signal detection for wireless communications • Tree-structured EP on loopy graphs • Conclusions

  12. x1 x2 x1 x2 y1 y2 y1 y2 x1 x2 x1 x2 y1 y2 y1 y2 EP on Dynamic Systems

  13. Object Tracking Guess the position of an object given noisy observations Object

  14. x1 x2 xT y1 y2 yT Bayesian Network e.g. (random walk) want distribution of x’s given y’s

  15. Approximation (proportional) Factorized and Gaussian in x

  16. xt yt Message Interpretation = (forward msg)(observation)(backward msg) Forward Message Backward Message Observation Message

  17. EP on Dynamic Systems • Filtering: t = 1, …, T • Incorporate forward message • Initialize observation message • Smoothing: t = T, …, 1 • Incorporate the backward message • Compute the leave-one-out approximation by dividing out the old observation messages • Re-approximate the new observation messages • Re-filtering: t = 1, …, T • Incorporate forward and observation messages

  18. Extension of EP • Instead of matching moments, use any method for approximate filtering. • Examples: Extended Kalman filter, statistical linearization, unscented filter • All methods can be interpreted as finding linear/Gaussian approximations to original terms

  19. Example: Poisson Tracking • is an integer valued Poisson variate with mean

  20. Poisson Tracking Model

  21. Approximate Observation Message • is not Gaussian • Moments of x not analytic • Two approaches: • Gauss-Hermite quadrature for moments • Statistical linearization instead of moment-matching • Both work well

  22. EP Accuracy Improves Significantly in only a few Iterations

  23. Approximate vs. Exact Posterior

  24. EP vs. Monte Carlo: Accuracy Mean Variance

  25. Accuracy/Efficiency Tradeoff

  26. EP for Digital Wireless Communication • Signal detection problem • Transmitted signal st = • vary to encode each symbol • Complex representation: Im Re

  27. Binary Symbols, Gaussian Noise • Symbols are 1 and –1 (in complex plane) • Received signal yt = • Optimal detection is easy

  28. Fading Channel • Channel systematically changes amplitude and phase: • changes over time

  29. Benchmark: Differential Detection • Classical technique • Use previous observation to estimate state • Binary symbols only

  30. s2 sT s1 y1 yT y2 x1 xT x2 Bayesian network for Signal Detection

  31. On-line EP Joint Signal Detector and Channel Estimation • Iterate over the last observations • Observations before act as prior for the current estimation

  32. Computational Complexity • Expectation propagation O(nLd2) • Stochastic mixture of Kalman filters O(LMd2) • Rao-blackwised paricle smoothers O(LMNd2) • n: Number of EP iterations (Typically, 4 or 5) • d: Dimension of the parameter vector • L: Smooth window length • M: Number of samples in filtering • N: Number of samples in smoothing

  33. Experimental Results (Chen, Wang, Liu 2000) EP outperforms particle smoothers in efficiency with comparable accuracy.

  34. e2 eT e1 y1 yT y2 x1 xT x2 Bayesian Networks for Adaptive Decoding The information bits et are coded by a convolutional error-correcting encoder.

  35. EP Outperforms Viterbi Decoding

  36. Outline • Background • Expectation Propagation (EP) on dynamic systems • Poisson tracking • Signal detection for wireless communications • Tree-structured EP on loopy graphs • Conclusions

  37. x1 x2 x1 x2 y1 y2 y1 y2 x1 x2 x1 x2 y1 y2 y1 y2 EP on Boltzman machines

  38. X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 Inference on Grids Problem: estimate marginal distributions of the variables indexed by the nodes in a loopy graph, e.g., p(xi), i = 1, . . . , 16.

  39. Boltzmann Machines Joint distribution is product of pair potentials: Want to approximate by a simpler distribution

  40. BP vs. EP EP BP

  41. Junction Tree Representation p(x) q(x) Junction tree

  42. Approximating an Edge by a Tree Each potential f ain p is projected onto the tree-structure of q Correlations are not lost, but projected onto the tree

  43. Moment Matching • Match single and pairwise marginals of • Reduces to exact inference on single loops • Use cutset conditioning and

  44. Local Propagation • Original EP: globally propagate evidence to the whole tree • Problem: Computationally expensive • Exploit the junction tree representation: only locally propagate evidence within the minimal subtree that is directly connected to the off-tree edge. • Reduce computational complexity • Save memory

  45. x5 x7 x1 x2 x1 x2 x5 x7 x1 x2 x1 x2 x5 x7 x1 x4 x3 x5 x1 x3 x1 x4 x3 x5 x3 x6 x1 x3 x3 x6 x1 x3 x1 x3 x1 x4 x3 x5 x3 x6 x1 x4 x3 x4 x3 x4 Global propagation Local propagation

  46. 4-node Graph TreeEP = the proposed method, BP = loopy belief propagation, GBP = generalized belief propagation on triangles, MF = mean-field, TreeVB =variational tree.

  47. Fully-connected graphs • Results are averaged over 10 graphs with randomly generated potentials • TreeEP performs the same or better than all other methods in both accuracy and efficiency!

  48. 8x8 grids, 10 trials

  49. TreeEP versus BP and GBP • TreeEP is always more accurate than BP and is often faster • TreeEP is much more efficient than GBP and more accurate on some problems • TreeEP converges more often than BP and GBP

  50. Outline • Background • Expectation Propagation (EP) on dynamic systems • Poisson tracking • Signal detection for wireless communications • Tree-structured EP on loopy graphs • Conclusions

More Related