Generalized Belief Propagation for Gaussian Graphical Model in probabilistic image processing

1. Generalized Belief Propagation for Gaussian Graphical Model in probabilistic image processing Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/ Reference K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A: Math & Gen., 37, 8675 (2004). SPDSA2005 (Roma)

2. Contents Introduction Loopy Belief Propagation Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks SPDSA2005 (Roma)

3. Noise Transmission Original Image Degraded Image Bayesian Image Analysis Graphical Model with Loops =Spin System on Square Lattice Bayesian Image Analysis + Belief Propagation → Probabilistic Image Processing SPDSA2005 (Roma)

4. Probabilistic model with no loop Belief Propagation Belief Propagation = Transfer Matrix (Lauritzen, Pearl) Probabilistic model with some loops Approximation→Loopy Belief Propagation (Yedidia, Freeman, Weiss) Generalized Belief Propagation Loopy Belief Propagation (LBP)= Bethe Approximation Generalized Belief Propagation (GBP) = Cluster Variation Method • How is the accuracy of LBP and GBP? SPDSA2005 (Roma)

5. Gaussian Graphical Model SPDSA2005 (Roma)

6. Probabilistic Image Processing by Gaussian Graphical Model and Generalized Belief Propagation • How can we construct a probabilistic image processing algorithm by using Loopy Belief Propagation and Generalized Belief Propagation? • How is the accuracy of Loopy Belief Propagation and Generalized Belief Propagation? • In order to clarify both questions, we assume the Gaussian graphical model as a posterior probabilistic model SPDSA2005 (Roma)

7. Contents Introduction Loopy Belief Propagation Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks SPDSA2005 (Roma)

8. Kullback-Leibler Divergence of Gaussian Graphical Model Entropy Term SPDSA2005 (Roma)

9. Loopy Belief Propagation Trial Function Tractable Form SPDSA2005 (Roma)

10. Loopy Belief Propagation Trial Function Marginal Distribution of GGM is also GGM SPDSA2005 (Roma)

11. Loopy Belief Propagation Bethe Free Energy in GGM SPDSA2005 (Roma)

12. Loopy Belief Propagation m is exact SPDSA2005 (Roma)

13. Fixed Point Equation Iteration Procedure Iteration SPDSA2005 (Roma)

14. Loopy Belief Propagation and TAP Free Energy Loopy Belief Propagation TAP Free Energy Mean Field Free Energy SPDSA2005 (Roma)

15. Contents Introduction Loopy Belief Propagation Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks SPDSA2005 (Roma)

16. 1 2 1 2 1 2 3 4 3 4 3 4 Generalized Belief Propagation Cluster: Set of nodes Every subcluster of the element of B does not belong to B. Example: System consisting of 4 nodes SPDSA2005 (Roma)

17. 1 2 3 4 5 6 7 8 9 5 2 1 4 5 3 6 2 2 1 2 3 4 1 3 6 5 2 6 5 8 9 8 7 4 5 5 4 7 8 9 6 5 8 7 4 8 6 5 9 Selection of B in LBP and GBP LBP (Bethe Approx.） GBP (Square Approx. in CVM) SPDSA2005 (Roma)

18. Selection of B and C in Loopy Belief Propagation LBP (Bethe Approx.） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters SPDSA2005 (Roma)

19. Selection of B and C in Generalized Belief Propagation GBP (Square Approximation in CVM） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters SPDSA2005 (Roma)

20. Generalized Belief Propagation Trial Function Marginal Distribution of GGM is also GGM SPDSA2005 (Roma)

21. Generalized Belief Propagation m is exact SPDSA2005 (Roma)

22. Contents Introduction Loopy Belief Propagation Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks SPDSA2005 (Roma)

23. Noise Transmission Bayesian Image Analysis Original Image Degraded Image SPDSA2005 (Roma)

24. Degradation Process Bayesian Image Analysis Additive White Gaussian Noise Transmission Original Image Degraded Image SPDSA2005 (Roma)

25. Standard Images A Priori Probability Bayesian Image Analysis Generate Similar? SPDSA2005 (Roma)

26. Original Image f Degraded Image g Bayesian Image Analysis A Posteriori Probability Gaussian Graphical Model SPDSA2005 (Roma)

27. Bayesian Image Analysis Degraded Image A Priori Probability Degraded Image Original Image Pixels A Posteriori Probability SPDSA2005 (Roma)

28. Hyperparameter Determination by Maximization of Marginal Likelihood Marginalization Degraded Image Original Image Marginal Likelihood SPDSA2005 (Roma)

29. Marginal Likelihood Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function Incomplete Data Equivalent SPDSA2005 (Roma)

30. EM Algorithm Iterate the following EM-steps until convergence: Marginal Likelihood Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977). SPDSA2005 (Roma)

31. Image Restoration • The original image is generated from the prior probability. (Hyperparameters: Maximization of Marginal Likelihood) Degraded Image Original Image Loopy Belief Propagation Mean-Field Method Exact Result SPDSA2005 (Roma)

32. Numerical Experiments of Logarithm of Marginal Likelihood • The original image is generated from the prior probability. (Hyperparameters: Maximization of Marginal Likelihood) Degraded Image Original Image MFA -5.0 -5.0 MFA LPB -5.5 Exact LPB Exact -5.5 -6.0 10 20 30 40 50 60 0 0.0010 0.0020 Mean-Field Method Loopy Belief Propagation Exact Result SPDSA2005 (Roma)

33. 0.002 Exact LBP MF 0.001 LPB Exact MFA 0 100 0 50 Numerical Experiments of Logarithm of Marginal Likelihood EM Algorithm with Belief Propagation Original Image Degraded Image SPDSA2005 (Roma)

34. Image Restoration by Gaussian Graphical Model EM Algorithm with Belief Propagation Original Image Degraded Image MSE: 1512 MSE: 1529 SPDSA2005 (Roma)

35. Image Restoration by Gaussian Graphical Model Original Image Degraded Image Mean Field Method MSE:611 MSE: 1512 LBP TAP GBP Exact Solution MSE:327 MSE:320 MSE: 315 MSE:315 SPDSA2005 (Roma)

36. Image Restoration by Gaussian Graphical Model Original Image Degraded Image Mean Field Method MSE: 1529 MSE: 565 LBP TAP GBP Exact Solution MSE:260 MSE:248 MSE:236 MSE:236 SPDSA2005 (Roma)

37. Image Restoration by Gaussian Graphical Model SPDSA2005 (Roma)

38. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (3x3) Lowpass (5x5) Median (5x5) Wiener SPDSA2005 (Roma)

39. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (5x5) Lowpass (5x5) Median (5x5) Wiener SPDSA2005 (Roma)

40. Contents Introduction Loopy Belief Propagation Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks SPDSA2005 (Roma)

41. Summary • Generalized Belief Propagation for Gaussian Graphical Model • Accuracy of Generalized Belief Propagation • Derivation of TAP Free Energy for Gaussian Graphical Model by Perturbation Expansion of Bethe Approximation SPDSA2005 (Roma)

42. Future Problem • Hyperparameter Estimation by TAP Free Energy is better than by Loopy Belief Propagation. • Effectiveness of Higher Order Terms of TAP Free Energy for Hyperparameter Estimation by means of Marginal Likelihood in Bayesian Image Analysis. Mean Field Free Energy TAP Free Energy SPDSA2005 (Roma)