Probabilistic image processing and Bayesian network

1. Probabilistic image processing and Bayesian network Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ • References • K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002). • K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004). RC2005 (19 July, 2005, Sendai)

2. Bayesian Network Bayes Formula Probabilistic Model Probabilistic Information Processing Bayesian Network and Belief Propagation Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). RC2005 (19 July, 2005, Sendai)

3. Formulation of Belief Propagation • Link between belief propagation and statistical mechanics. Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett.44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory andPractice (MIT Press, 2001). • Generalized belief propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Information geometrical interpretation of belief propagation • S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004). RC2005 (19 July, 2005, Sendai)

4. Application of Belief Propagation • Image Processing K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002). • Low Density Parity Check Codes Y. Kabashima and D. Saad: Statistical mechanics oflow-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004). • CDMA Multiuser Detection Algorithm Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003). T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005). • Satisfability Problem O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265(2001). M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002). RC2005 (19 July, 2005, Sendai)

5. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Image Segmentation • Concluding Remarks RC2005 (19 July, 2005, Sendai)

6. Belief Propagation • How should we treat the calculation of the summation over 2N configurations. It is very hard to calculate exactly except some special cases. • Formulation for approximate algorithm • Accuracy of the approximate algorithm RC2005 (19 July, 2005, Sendai)

7. Tractable Model • Probabilistic models with no loop are tractable. Factorizable • Probabilistic models with loop are not tractable. Not Factorizable RC2005 (19 July, 2005, Sendai)

8. Probabilistic Model on a Graph with Loops Marginal Probability RC2005 (19 July, 2005, Sendai)

9. 3 1 4 2 5 Message Passing Rule of Belief Propagation Fixed Point Equations for Massage RC2005 (19 July, 2005, Sendai)

10. 3 1 4 2 5 Approximate Representation of Marginal Probability Fixed Point Equations for Messages RC2005 (19 July, 2005, Sendai)

11. Fixed Point Equation Fixed Point Equation and Iterative Method Iterative Method RC2005 (19 July, 2005, Sendai)

12. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Image Segmentation • Concluding Remarks RC2005 (19 July, 2005, Sendai)

13. Noise Transmission Bayesian Image Analysis Original Image Degraded Image RC2005 (19 July, 2005, Sendai)

14. Degradation Process Bayesian Image Analysis Additive White Gaussian Noise Transmission Original Image Degraded Image RC2005 (19 July, 2005, Sendai)

15. Standard Images A Priori Probability Bayesian Image Analysis Generate Similar? RC2005 (19 July, 2005, Sendai)

16. A Posteriori Probability Bayesian Image Analysis Gaussian Graphical Model RC2005 (19 July, 2005, Sendai)

17. Bayesian Image Analysis Degraded Image A Priori Probability Degraded Image Original Image Pixels A Posteriori Probability RC2005 (19 July, 2005, Sendai)

18. Hyperparameter Determination by Maximization of Marginal Likelihood Marginalization Degraded Image Original Image Marginal Likelihood RC2005 (19 July, 2005, Sendai)

19. Marginal Likelihood Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function Incomplete Data Equivalent RC2005 (19 July, 2005, Sendai)

20. EM Algorithm Iterate the following EM-steps until convergence: Marginal Likelihood Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977). RC2005 (19 July, 2005, Sendai)

21. Original Signal 200 100 0 0 255 127 Degraded Signal 200 100 0 0 255 127 Estimated Signal 200 100 0 0 255 127 One-Dimensional Signal EM Algorithm RC2005 (19 July, 2005, Sendai)

22. Image Restoration by Gaussian Graphical Model EM Algorithm with Belief Propagation Original Image Degraded Image MSE: 1512 MSE: 1529 RC2005 (19 July, 2005, Sendai)

23. Exact Results of Gaussian Graphical Model Multi-dimensional Gauss integral formula RC2005 (19 July, 2005, Sendai)

24. Comparison of Belief Propagation with Exact Results in Gaussian Graphical Model RC2005 (19 July, 2005, Sendai)

25. Image Restoration by Gaussian Graphical Model Exact Original Image Degraded Image Belief Propagation MSE:315 MSE: 325 MSE: 1512 Lowpass Filter Median Filter Wiener Filter MSE: 411 MSE: 545 MSE: 447 RC2005 (19 July, 2005, Sendai)

26. Image Restoration by Gaussian Graphical Model Belief Propagation Original Image Degraded Image Exact MSE236 MSE: 260 MSE: 1529 Median Filter Wiener Filter Lowpass Filter MSE: 224 MSE: 372 MSE: 244 RC2005 (19 July, 2005, Sendai)

27. Extension of Belief Propagation • Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Generalized belief propagation is equivalent to the cluster variation method in statistical mechanics R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957). RC2005 (19 July, 2005, Sendai)

28. Image Restoration by Gaussian Graphical Model RC2005 (19 July, 2005, Sendai)

29. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (3x3) Lowpass (5x5) Median (5x5) Wiener RC2005 (19 July, 2005, Sendai)

30. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (5x5) Lowpass (5x5) Median (5x5) Wiener RC2005 (19 July, 2005, Sendai)

31. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Image Segmentation • Concluding Remarks RC2005 (19 July, 2005, Sendai)

32. Image Segmentation by Gauss Mixture Model Gauss Mixture Model RC2005 (19 July, 2005, Sendai)

33. Image Segmentation by Combining Gauss Mixture Model with Potts Model Belief Propagation Potts Model RC2005 (19 July, 2005, Sendai)

34. Image Segmentation Gauss Mixture Model Gauss Mixture Model and Potts Model Original Image Histogram Belief Propagation RC2005 (19 July, 2005, Sendai)

35. Motion Detection a Segmentation b Detection AND c Segmentation Gauss Mixture Model and Potts Model with Belief Propagation RC2005 (19 July, 2005, Sendai)

36. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Image Segmentation • Concluding Remarks RC2005 (19 July, 2005, Sendai)

37. Summary • Formulation of belief propagation • Accuracy of belief propagation in Bayesian image analysis by means of Gaussian graphical model (Comparison between the belief propagation and exact calculation) • Some applications of Bayesian image analysis and belief propagation RC2005 (19 July, 2005, Sendai)

38. Related Problem Statistical Performance Spin Glass Theory H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001. RC2005 (19 July, 2005, Sendai)

39. 確率的情報処理の動向 • 田中和之・樺島祥介編著, “ミニ特集/ベイズ統計・統計力学と情報処理”, 計測と制御 2003年8月号． • 田中和之，田中利幸，渡辺治 他著，“連載/確率的情報処理と統計力学 ～様々なアプローチとそのチュートリアル～”，数理科学2004年11月号から開始． • 田中和之，岡田真人，堀口剛 他著，“小特集/確率を手なづける秘伝の計算技法 ～古くて新しい確率・統計モデルのパラダイム～”，電子情報通信学会誌 2005年9月号 RC2005 (19 July, 2005, Sendai)