Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing

1. Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/ National Tsin Hua University, Taiwan

2. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

3. Mathematical expression of uncertainty =>Probability and Statistics Computational model for information processing in data with uncertainty Probabilistic model with graphical structure （Bayesian network） modeling Probabilistic Inference Node is random variable. Arrow is conditional probability. Inference system for data with uncertainty Graph with cycles Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness. Important aspect National Tsin Hua University, Taiwan

4. Bayes Formula Computational Model for Probabilistic Information Processing Probabilistic Information Processing Probabilistic Model Algorithm • Monte Carlo Method • Markov Chain Monte Carlo Method • Randomized Algorithm • Approximate Method • Belief Propagation • Variational Bayes Method • Expectation Propagation Randomness and Approximation National Tsin Hua University, Taiwan

5. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

6. a b Joint Probability and Conditional Probability Random Variable Probability Distribution Probability of Event A=a Joint Probability of Events A=a and B=b State Variable Conditional Probability of Event A=a when Event B=b has happened. Joint Probability Distribution National Tsin Hua University, Taiwan

7. a b Joint Probability and Independency of Events Events A and B are independent of each other In this case, the conditional probability can be expressed as a b National Tsin Hua University, Taiwan

8. Marginal Probability Let us suppose that the sample space Wis expressed by Ω= (A=0)∪ (A=1) ∪…∪ (A=M-1) where every pair of events is exclusive of each other. Graph with Two Nodes and One Edge a b Marginal Probability of Event B=b in Joint Probability Pr{A=a,B=b} Message a b = Simplified Notation Marginalize Summation over all the possible events in which every pair of events are exclusive of each other. National Tsin Hua University, Taiwan

9. Marginal Probabiilty of High-Dimentional Joint Probabilty Marginalization with respect to a, c and d a b a b Hyperedge Message d c c d Hypergraph Marginalization with respect to c and d a b a b Hyperedge Message d c c d Hypergraph National Tsin Hua University, Taiwan

10. a b Bayes Formulas Bayesian Network Posterior Probability Prior Probability Marginal Likelihood National Tsin Hua University, Taiwan

11. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

12. Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability. Image Restoration by Probabilistic Model Noise Transmission Original Image Degraded Image Bayes Formula National Tsin Hua University, Taiwan

13. xi xj Prior Probability in Probabilistic Image Processing x1 x2 x3 x4 xi xj x5 x6 x7 x8 State Variable of Light Intensity at i-th Pixel in Original Image xi x9 x10 x11 x12 National Tsin Hua University, Taiwan

14. Conditional Probability of Degradation Process Additive White Gaussian Noise xi yi State Variable of Light Intensity at i-th Pixel in Original Image xi y1 y2 y3 y4 X1 X2 X3 X4 State Variable of Light Intensity at i-th Pixel in Original Image y5 y6 y7 y8 yi X5 X6 X7 X8 y9 y10 y11 y12 xi yi X9 X10 X11 X12 National Tsin Hua University, Taiwan

15. Bayesian Image Analysis Original Image Degraded Image Prior Probability Degradation Process Posterior Probability Image Processing is reduced to computations of avereages, variance at each pixel and covariances of each pair of neghbouring pixels National Tsin Hua University, Taiwan

16. Hyperparameters a, s are determinedso as to maximize the marginal likelihood Pr{Y=y|a,s} with respect to a, s. Statistical Estimation of Hyperparameters EM (Expectation Maximization) Algorithm Degraded Image Original Image Marginalized with respect to X Marginal Likelihood National Tsin Hua University, Taiwan

17. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

18. Gaussian Graphical Model(Gauss Markov Random Fields) Multidimensional Gauss Integral Formulas Maximum Likelihood Estimation  EM Algorithm National Tsin Hua University, Taiwan

19. Original Signal 200 100 0 0 255 127 Degraded Signal 200 100 0 0 255 127 Estimated Signal 200 100 0 0 255 127 One-Dimensional Signal Processing EM Algorithm National Tsin Hua University, Taiwan

20. Iteration Procedure of EM algorithm in Gaussian Graphical Model Bayesian Image Analysis by Gaussian Graphical Model EM National Tsin Hua University, Taiwan

21. Image Restoration by Gaussian Graphical Model and Conventional Filters Degraded Image (s=40) Original Image V:Set of all the pixels Gaussian Graphical Model (3x3) Lowpass (5x5) Median National Tsin Hua University, Taiwan

22. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

23. What is an important point in computational complexity? • How should we treat the calculation of the summation over 2N configuration? If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. N fold loops • Markov Chain Monte Carlo Method • Belief Propagation Method This Talk National Tsin Hua University, Taiwan

24. Strategy of Approximate Algorithm in Probabilistic Information Processing It is very hard to compute marginal probabilities exactly except some tractable cases. • What is the tractable cases in which marginal probabilities can be computed exactly? • Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases? National Tsin Hua University, Taiwan

25. Graphical Representations of Tractable Models National Tsin Hua University, Taiwan

26. Graphical Representations of Tractable Models a b c d e X b c d e a b b c d e a b a b a b c d e National Tsin Hua University, Taiwan

27. Graphical Representations of Tractable Models a b c d e X a b c c d e X a b c c d e b c b c d e National Tsin Hua University, Taiwan

28. Graphical Representations of Tractable Models a b c d e a b c d e b c d e c d e d e National Tsin Hua University, Taiwan

29. Graphical Representations of Tractable Models a d a d c c e e b f b f a d d c e c e b f f d e c e f f National Tsin Hua University, Taiwan

30. Belief Propagation for Probabilistic Model on Square Grid Graph E: Set of all the links National Tsin Hua University, Taiwan

31. 2 2 Marginal Probability National Tsin Hua University, Taiwan

32. 1 2 1 2 Marginal Probability National Tsin Hua University, Taiwan

33. Belief Propagation 3 1 2 2 1 4 5 Message Update Rule 3 8 8 2 2 1 1 7 7 4 6 6 5 National Tsin Hua University, Taiwan

34. 3 4 1 2 5 Belief Propagation on Graph with Cycles 3 1 2 2 1 4 5 Simultaneous Fixed Point Equations of Messages Average, variances and covariances can be expressed in terms of messages. National Tsin Hua University, Taiwan

35. Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method National Tsin Hua University, Taiwan

36. Fundamental Structures of Belief Propagation in Probabilistic Image Processing Three Inputs and One Output Message Passing Rules National Tsin Hua University, Taiwan

37. ３ ４ １ ２ ５ Belief Propagation and EM Algorithm Update Rule of BP Input BP EM Output National Tsin Hua University, Taiwan

38. Maximization of Marginal Likelihood by EM Algorithm Exact Loopy Belief Propagation National Tsin Hua University, Taiwan

39. Image Restoration by Gaussian Graphical Model Belief Propagation Original Image Degraded Image Exact MSE:325 MSE:315 MSE: 1512 Lowpass Filter Median Filter Wiener Filter MSE: 411 MSE: 545 MSE: 447 National Tsin Hua University, Taiwan

40. Digital Images Inpainting based on MRF Markov Random Fields Output Input M. Yasuda, J. Ohkubo and K. Tanaka: Proceedings of CIMCA&IAWTIC2005. National Tsin Hua University, Taiwan

41. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

42. Belief Propagation for Bayesian Networks National Tsin Hua University, Taiwan

43. Factor Graph Representations of Bayesian Networks and Belief Propagations National Tsin Hua University, Taiwan

44. Error Correcting Code Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004. error 010 000001111100000 001001011100001 code decode majority rule 010 0 1 0 Error Correcting Codes Parity Check Code Turbo Code, Low Density Parity Check (LDPC) Code High Performance Decoding Algorithm National Tsin Hua University, Taiwan

45. Error Correcting Codes and Belief Propagation Received Word 1 1 0 Binary Symmetric Channel 0 1 0 Code Word National Tsin Hua University, Taiwan

46. Error Correcting Codes and Belief Propagation Fundamental Concept for Turbo Codes and LDPC Codes National Tsin Hua University, Taiwan

47. Satisfactory Problem (3-SAT) National Tsin Hua University, Taiwan

48. Contents • Introduction • Bayesian Statistics • Probabilistic Image Processing • Gaussian Graphical Model • Belief Propagation • Statistical Performance Analysis • Various Applications of Probabilistic Information Processing • Summary National Tsin Hua University, Taiwan

49. Summary • Fundamental Structures of Bayesian modeling have been introduced. • Formulation of probabilistic image processing algorithms by means of loopy belief propagation has been summarized. • Various applications of Bayesian Network Systems have been reviewed. National Tsin Hua University, Taiwan

50. References • K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007. • M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007. • K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008 • K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009 • M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009. • S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010. National Tsin Hua University, Taiwan