Probabilistic image processing and Bayesian network

1. Probabilistic image processing and Bayesian network Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ • References • K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002). • K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004). CISJ2005

2. Bayesian Network Bayes Formula Probabilistic Model Probabilistic Information Processing Bayesian Network and Belief Propagation Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). CISJ2005

3. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Concluding Remarks CISJ2005

4. Belief Propagation • How should we treat the calculation of the summation over 2N configuration? It is very hard to calculate exactly except some special cases. • Formulation for approximate algorithm • Accuracy of the approximate algorithm CISJ2005

5. Tractable Model • Probabilistic models with no loop are tractable. Factorizable • Probabilistic models with loop are not tractable. Not Factorizable CISJ2005

6. 3 4 1 2 6 5 Probabilistic model on a graph with no loop Marginal probability of the node 2 CISJ2005

7. 3 4 2 1 6 5 Probabilistic model on a graph with no loop Marginal probability can be expressed in terms of the product of messages from all the neighbouring nodes of node 2. Message from the node 1 to the node 2 can be expressed in terms of the product of message from all the neighbouring nodes of the node 1 except one from the node 2. CISJ2005

8. Probabilistic Model on a Graph with Loops Marginal Probability CISJ2005

9. 3 1 4 2 3 8 1 2 5 4 7 5 6 Belief Propagation Message Update Rule CISJ2005

10. 3 1 4 2 5 Message Passing Rule of Belief Propagation Fixed Point Equations for Massage CISJ2005

11. Fixed Point Equation Fixed Point Equation and Iterative Method Iterative Method CISJ2005

12. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Concluding Remarks CISJ2005

13. Noise Transmission Bayesian Image Analysis Original Image Degraded Image CISJ2005

14. Degradation Process Bayesian Image Analysis Additive White Gaussian Noise Transmission Original Image Degraded Image CISJ2005

15. Standard Images A Priori Probability Bayesian Image Analysis Generate Similar? CISJ2005

16. A Posteriori Probability Bayesian Image Analysis Gaussian Graphical Model CISJ2005

17. Bayesian Image Analysis Degraded Image A Priori Probability Degraded Image Original Image Pixels A Posteriori Probability CISJ2005

18. Hyperparameter Determination by Maximization of Marginal Likelihood Marginalization Degraded Image Original Image Marginal Likelihood CISJ2005

19. Marginal Likelihood Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function Incomplete Data Equivalent CISJ2005

20. EM Algorithm Iterate the following EM-steps until convergence: Marginal Likelihood Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Q-Function A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977). CISJ2005

21. Original Signal 200 100 0 0 255 127 Degraded Signal 200 100 0 0 255 127 Estimated Signal 200 100 0 0 255 127 One-Dimensional Signal EM Algorithm CISJ2005

22. Image Restoration by Gaussian Graphical Model EM Algorithm with Belief Propagation Original Image Degraded Image MSE: 1512 MSE: 1529 CISJ2005

23. Exact Results of Gaussian Graphical Model Multi-dimensional Gauss integral formula CISJ2005

24. Comparison of Belief Propagation with Exact Results in Gaussian Graphical Model CISJ2005

25. Image Restoration by Gaussian Graphical Model Exact Original Image Degraded Image Belief Propagation MSE:315 MSE: 325 MSE: 1512 Lowpass Filter Median Filter Wiener Filter MSE: 411 MSE: 545 MSE: 447 CISJ2005

26. Image Restoration by Gaussian Graphical Model Belief Propagation Original Image Degraded Image Exact MSE236 MSE: 260 MSE: 1529 Median Filter Wiener Filter Lowpass Filter MSE: 224 MSE: 372 MSE: 244 CISJ2005

27. Extension of Belief Propagation • Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Generalized belief propagation is equivalent to the cluster variation method in statistical mechanics R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957). CISJ2005

28. Image Restoration by Gaussian Graphical Model CISJ2005

29. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (3x3) Lowpass (5x5) Median (5x5) Wiener CISJ2005

30. Image Restoration by Gaussian Graphical Model and Conventional Filters GBP (5x5) Lowpass (5x5) Median (5x5) Wiener CISJ2005

31. Contents • Introduction • Belief Propagation • Bayesian Image Analysis and Gaussian Graphical Model • Concluding Remarks CISJ2005

32. Summary • Formulation of belief propagation • Accuracy of belief propagation in Bayesian image analysis by means of Gaussian graphical model (Comparison between the belief propagation and exact calculation) CISJ2005