1 / 51

Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing

Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing. Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/. Contents. Introduction Bayesian Statistics

sonel
Download Presentation

Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/ National Tsin Hua University, Taiwan

  2. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  3. Mathematical expression of uncertainty =>Probability and Statistics Computational model for information processing in data with uncertainty Probabilistic model with graphical structure (Bayesian network) modeling Probabilistic Inference Node is random variable. Arrow is conditional probability. Inference system for data with uncertainty Graph with cycles Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness. Important aspect National Tsin Hua University, Taiwan

  4. Bayes Formula Computational Model for Probabilistic Information Processing Probabilistic Information Processing Probabilistic Model Algorithm • Monte Carlo Method • Markov Chain Monte Carlo Method • Randomized Algorithm • Approximate Method • Belief Propagation • Variational Bayes Method • Expectation Propagation Randomness and Approximation National Tsin Hua University, Taiwan

  5. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  6. a b Joint Probability and Conditional Probability Random Variable Probability Distribution Probability of Event A=a Joint Probability of Events A=a and B=b State Variable Conditional Probability of Event A=a when Event B=b has happened. Joint Probability Distribution National Tsin Hua University, Taiwan

  7. a b Joint Probability and Independency of Events Events A and B are independent of each other In this case, the conditional probability can be expressed as a b National Tsin Hua University, Taiwan

  8. Marginal Probability Let us suppose that the sample space Wis expressed by Ω= (A=0)∪ (A=1) ∪…∪ (A=M-1) where every pair of events is exclusive of each other. Graph with Two Nodes and One Edge a b Marginal Probability of Event B=b in Joint Probability Pr{A=a,B=b} Message a b = Simplified Notation Marginalize Summation over all the possible events in which every pair of events are exclusive of each other. National Tsin Hua University, Taiwan

  9. Marginal Probabiilty of High-Dimentional Joint Probabilty Marginalization with respect to a, c and d a b a b Hyperedge Message d c c d Hypergraph Marginalization with respect to c and d a b a b Hyperedge Message d c c d Hypergraph National Tsin Hua University, Taiwan

  10. a b Bayes Formulas Bayesian Network Posterior Probability Prior Probability Marginal Likelihood National Tsin Hua University, Taiwan

  11. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  12. Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability. Image Restoration by Probabilistic Model Noise Transmission Original Image Degraded Image Bayes Formula National Tsin Hua University, Taiwan

  13. xi xj Prior Probability in Probabilistic Image Processing x1 x2 x3 x4 xi xj x5 x6 x7 x8 State Variable of Light Intensity at i-th Pixel in Original Image xi x9 x10 x11 x12 National Tsin Hua University, Taiwan

  14. Conditional Probability of Degradation Process Additive White Gaussian Noise xi yi State Variable of Light Intensity at i-th Pixel in Original Image xi y1 y2 y3 y4 X1 X2 X3 X4 State Variable of Light Intensity at i-th Pixel in Original Image y5 y6 y7 y8 yi X5 X6 X7 X8 y9 y10 y11 y12 xi yi X9 X10 X11 X12 National Tsin Hua University, Taiwan

  15. Bayesian Image Analysis Original Image Degraded Image Prior Probability Degradation Process Posterior Probability Image Processing is reduced to computations of avereages, variance at each pixel and covariances of each pair of neghbouring pixels National Tsin Hua University, Taiwan

  16. Hyperparameters a, s are determinedso as to maximize the marginal likelihood Pr{Y=y|a,s} with respect to a, s. Statistical Estimation of Hyperparameters EM (Expectation Maximization) Algorithm Degraded Image Original Image Marginalized with respect to X Marginal Likelihood National Tsin Hua University, Taiwan

  17. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  18. Gaussian Graphical Model(Gauss Markov Random Fields) Multidimensional Gauss Integral Formulas Maximum Likelihood Estimation  EM Algorithm National Tsin Hua University, Taiwan

  19. Original Signal 200 100 0 0 255 127 Degraded Signal 200 100 0 0 255 127 Estimated Signal 200 100 0 0 255 127 One-Dimensional Signal Processing EM Algorithm National Tsin Hua University, Taiwan

  20. Iteration Procedure of EM algorithm in Gaussian Graphical Model Bayesian Image Analysis by Gaussian Graphical Model EM National Tsin Hua University, Taiwan

  21. Image Restoration by Gaussian Graphical Model and Conventional Filters Degraded Image (s=40) Original Image V:Set of all the pixels Gaussian Graphical Model (3x3) Lowpass (5x5) Median National Tsin Hua University, Taiwan

  22. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  23. What is an important point in computational complexity? • How should we treat the calculation of the summation over 2N configuration? If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. N fold loops • Markov Chain Monte Carlo Method • Belief Propagation Method This Talk National Tsin Hua University, Taiwan

  24. Strategy of Approximate Algorithm in Probabilistic Information Processing It is very hard to compute marginal probabilities exactly except some tractable cases. • What is the tractable cases in which marginal probabilities can be computed exactly? • Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases? National Tsin Hua University, Taiwan

  25. Graphical Representations of Tractable Models National Tsin Hua University, Taiwan

  26. Graphical Representations of Tractable Models a b c d e X b c d e a b b c d e a b a b a b c d e National Tsin Hua University, Taiwan

  27. Graphical Representations of Tractable Models a b c d e X a b c c d e X a b c c d e b c b c d e National Tsin Hua University, Taiwan

  28. Graphical Representations of Tractable Models a b c d e a b c d e b c d e c d e d e National Tsin Hua University, Taiwan

  29. Graphical Representations of Tractable Models a d a d c c e e b f b f a d d c e c e b f f d e c e f f National Tsin Hua University, Taiwan

  30. Belief Propagation for Probabilistic Model on Square Grid Graph E: Set of all the links National Tsin Hua University, Taiwan

  31. 2 2 Marginal Probability National Tsin Hua University, Taiwan

  32. 1 2 1 2 Marginal Probability National Tsin Hua University, Taiwan

  33. Belief Propagation 3 1 2 2 1 4 5 Message Update Rule 3 8 8 2 2 1 1 7 7 4 6 6 5 National Tsin Hua University, Taiwan

  34. 3 4 1 2 5 Belief Propagation on Graph with Cycles 3 1 2 2 1 4 5 Simultaneous Fixed Point Equations of Messages Average, variances and covariances can be expressed in terms of messages. National Tsin Hua University, Taiwan

  35. Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method National Tsin Hua University, Taiwan

  36. Fundamental Structures of Belief Propagation in Probabilistic Image Processing Three Inputs and One Output Message Passing Rules National Tsin Hua University, Taiwan

  37. 4 1 2 5 Belief Propagation and EM Algorithm Update Rule of BP Input BP EM Output National Tsin Hua University, Taiwan

  38. Maximization of Marginal Likelihood by EM Algorithm Exact Loopy Belief Propagation National Tsin Hua University, Taiwan

  39. Image Restoration by Gaussian Graphical Model Belief Propagation Original Image Degraded Image Exact MSE:325 MSE:315 MSE: 1512 Lowpass Filter Median Filter Wiener Filter MSE: 411 MSE: 545 MSE: 447 National Tsin Hua University, Taiwan

  40. Digital Images Inpainting based on MRF Markov Random Fields Output Input M. Yasuda, J. Ohkubo and K. Tanaka: Proceedings of CIMCA&IAWTIC2005. National Tsin Hua University, Taiwan

  41. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  42. Belief Propagation for Bayesian Networks National Tsin Hua University, Taiwan

  43. Factor Graph Representations of Bayesian Networks and Belief Propagations National Tsin Hua University, Taiwan

  44. Error Correcting Code Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004. error 010 000001111100000 001001011100001 code decode majority rule 010 0 1 0 Error Correcting Codes Parity Check Code Turbo Code, Low Density Parity Check (LDPC) Code High Performance Decoding Algorithm National Tsin Hua University, Taiwan

  45. Error Correcting Codes and Belief Propagation Received Word 1 1 0 Binary Symmetric Channel 0 1 0 Code Word National Tsin Hua University, Taiwan

  46. Error Correcting Codes and Belief Propagation Fundamental Concept for Turbo Codes and LDPC Codes National Tsin Hua University, Taiwan

  47. Satisfactory Problem (3-SAT) National Tsin Hua University, Taiwan

  48. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Statistical Performance Analysis • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

  49. Summary • Fundamental Structures of Bayesian modeling have been introduced. • Formulation of probabilistic image processing algorithms by means of loopy belief propagation has been summarized. • Various applications of Bayesian Network Systems have been reviewed. National Tsin Hua University, Taiwan

  50. References • K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007. • M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007. • K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008 • K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009 • M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009. • S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010. National Tsin Hua University, Taiwan

More Related