画像処理における確率伝搬法と EM アルゴリズムの統計的性能評価

1. 画像処理における確率伝搬法とEMアルゴリズムの統計的性能評価画像処理における確率伝搬法とEMアルゴリズムの統計的性能評価 東北大学大学院情報科学研究科 田中和之 http://www.smapip.is.tohoku.ac.jp/~kazu/ 共同研究者: D. M. Titterington (University of Glasgow) 皆川まりか (東北大) Reference 田中和之: ガウシアングラフィカルモデルにもとづく確率的情報処理における一般化された信念伝搬法, 電子情報通信学会論文誌 (D-II), Vol.J88-D-II, No.12, pp.2368-2379, 2005 Kyoto University

2. Contents Introduction Gaussian Graphical Model and EM Algorithm Loopy Belief Propagation Generalized Belief Propagation Concluding Remarks Kyoto University

3. MRF, Belief Propagation and Statistical Performance • Geman and Geman (1986): IEEE Transactions on PAMI • Image Processing for Markov Random Fields (MRF) (Simulated Annealing, Line Fields) • Tanaka and Morita (1995): Physics Letters A • Cluster Variation Method for MRF in Image Processing • Cluster Variation Method (CVM) • = Generalized Belief Propagation (GBP) • Nishimori and Wong (1999): Physical Review E • Statistical Performance Estimation for MRF • (Infinite Range Model and Replica Theory) Is it possible to estimate the performance of loopy belief propagation statistically? Kyoto University

4. Contents Introduction Gaussian Graphical Model and EM Algorithm Loopy Belief Propagation Generalized Belief Propagation Concluding Remarks Kyoto University

5. Noise Bayesian Image Restoration transmission Degraded Image Original Image Kyoto University

6. Bayes Formula and Probabilistic Image Processing Prior Probability Degradation Process Original Image Degraded Image Posterior Probability Pixel Kyoto University

7. Prior Probability in Probabilistic Image Processing Samples are generated by MCMC. Markov Chain Monte Carlo Method Kyoto University

8. Degradation Process Additive White Gaussian Noise Histogram of Gaussian Random Numbers Kyoto University

9. Degradation Process Degradation Process and Prior Prior Probability Density Function Posterior Probability Density Function Multi-Dimensional Gaussian Integral Formula Kyoto University

10. Statistical Performance by Sample Average Prior Probability Degradation Process Posterior Probability Kyoto University

11. Statistical Performance Analysis Prior Probability Degradation Process Posterior Probability Kyoto University

12. Statistical Performance Analysis Nishimori (2000) Multi-Dimentional Gaussian Integral Formula Kyoto University

13. Probabilistic Image Processing Posterior Probability Density Function Marginalized Marginal Likelihood Kyoto University

14. Marginalized Marginal Likelihood Marginal Likelihood in Probabilistic Image Processing Kyoto University

15. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Q-Function EM Algorithm Iterate the following EM-steps until convergence: A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977). Kyoto University

16. Pixel Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Q-Function Incomplete Data Equivalent Kyoto University

17. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Kyoto University

18. Statistical Behaviour of EM (Expectation Maximization) Algorithm Numerical Experiments for Standard Image Statistical Behaviour of EM Algorithm Kyoto University

19. Contents Introduction Gaussian Graphical Model and EM Algorithm Loopy Belief Propagation Generalized Belief Propagation Concluding Remarks Kyoto University

20. ３ ３ ４ １ ２ ４ １ ２ ５ ５ Belief Propagation and Markov Random Field Graphical Model with Cycles Marginal Probability Fixed Point Equation Kyoto University

21. Gaussian Graphical Model and Loopy Belief Propagation • Loopy Belief Propagation for Gaussian Graphical Model • Y. Weiss and W. T. Freeman, Correctness of belief propagation in Gaussian graphical models of arbitrary topology, Neural Computation, 13, 2173 (2001). • K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A, Math. & Gen., 37, 8675 (2004). • Dynamics of Algorithm in LBP? Statistical Analysis Kyoto University

22. Kullback-Leibler Divergence of Gaussian Graphical Model Entropy Term Kyoto University

23. Loopy Belief Propagation Trial Function Tractable Form Kyoto University

24. Loopy Belief Propagation Trial Function Marginal Distribution of GGM is also GGM Kyoto University

25. Loopy Belief Propagation Bethe Free Energy in GGM Kyoto University

26. ３ ４ １ ５ ２ Loopy Belief Propagation Vii andVij do not depend on pixel i and link ij Kyoto University

27. Iteration Procedure Fixed Point Equation Iteration Kyoto University

28. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Exact Loopy Belief Propagation Kyoto University

29. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Kyoto University

30. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Numerical Experiments for Standard Image Statistical Behaviour of EM Algorithm Kyoto University

31. Contents Introduction Gaussian Graphical Model and EM Algorithm Loopy Belief Propagation Generalized Belief Propagation Concluding Remarks Kyoto University

32. Generalized Belief Propagation • Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms J. S. Yedidia, W. T. Freeman and Y. Weiss: Transactions on Information Theory 2005. • Generalized Belief Propagation for Gaussian Graphical Model K. Tanaka: IEICE Transactions on Information and Systems 2005. Kyoto University

33. 1 2 1 2 1 2 3 4 3 4 3 4 Generalized Belief Propagation Cluster: Set of nodes Every subcluster of the element ofBdoes not belong toB. Example: System consisting of 4 nodes Kyoto University

34. 1 2 3 4 5 6 7 8 9 5 2 1 4 5 3 6 2 2 1 2 3 4 1 3 6 5 2 6 5 8 9 8 7 4 5 5 4 7 8 9 6 5 8 7 4 8 6 5 9 Selection ofBin LBP and GBP LBP (Bethe Approx.） GBP (Square Approx. in CVM) Kyoto University

35. Selection of B and C in Loopy Belief Propagation LBP (Bethe Approx.） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters Kyoto University

36. Selection of B and C in Generalized Belief Propagation GBP (Square Approximation in CVM） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters Kyoto University

37. Generalized Belief Propagation Trial Function Marginal Distribution of GGM is also GGM Kyoto University

38. ３ ４ １ ５ ２ Generalized Belief Propagation Kyoto University

39. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Exact Generalized Belief Propagation Loopy Belief Propagation Kyoto University

40. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Kyoto University

41. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Kyoto University

42. Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Numerical Experiments for Standard Image Statistical Behaviour of EM Algorithm Kyoto University

43. Image Restoration by Gaussian Graphical Model Original Image Degraded Image Mean Field Method MSE:604 MSE: 1511 LBP TAP GBP Exact Solution MSE:328 MSE:318 MSE: 314 MSE:314 Kyoto University

44. Image Restoration by Gaussian Graphical Model Original Image Degraded Image Mean Field Method MSE: 1529 MSE: 565 TAP GBP Exact Solution BP MSE:260 MSE:248 MSE:236 MSE:236 Kyoto University

45. Image Restoration by Gaussian Graphical Model Kyoto University

46. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (3x3) Lowpass (5x5) Median (5x5) Wiener Kyoto University

47. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (5x5) Lowpass (5x5) Median (5x5) Wiener Kyoto University

48. Contents Introduction Gaussian Graphical Model and EM Algorithm Loopy Belief Propagation Generalized Belief Propagation Concluding Remarks Kyoto University

49. Summary • Statistical Analysis of EM Algorithm in Generalized Belief Propagation for Gaussian Graphical Model Future Problems • General Scheme of Statistical Analysis for EM Algorithm with Generalized Belief Propagation. CVM for spin glass models may be useful. Kyoto University

50. Markov Chain Monte Carlo Method w(x(t+1)|x(t)) x(t) x(t+1) Kyoto University