Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/

1. Statistical Mechanical Approachto Probabilistic Information Processing:Cluster Variation Method and Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/ Collaborators Muneki Yasuda (GSIS, Tohoku University, Japan) Sun Kataoka (GSIS, Tohoku University, Japan) D. M. Titterington(Department of Statistics, University of Glasgow, UK) WPI, Tohoku University, Sendai

2. Outline • Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation • Bayesian Image Modeling by Generalized Sparse Prior • Noise Reductions by Generalized Sparse Prior • Concluding Remarks WPI, Tohoku University, Sendai

3. Bayesian Networks Markov Random Fields Probabilistic Model and Belief Propagation Bayes Formulas Probabilistic Models Probabilistic Information Processing Belief Propagation =Bethe Approximation i j Message =Effective Field V: Set of all the nodes (vertices) in graph G E: Set of all the links (edges) in graph G WPI, Tohoku University, Sendai

4. Mathematical Formulation of Belief Propagation • Similarity of Mathematical Structures between Mean Field Theory and Belief Propagation Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory andPractice (MIT Press, 2001). • Generalization of Belief Propagation S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Interpretations of Belief Propagation based on Information Geometry S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004). WPI, Tohoku University, Sendai

5. Generalized Extensions of Belief Propagation based on Cluster Variation Method • Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Cluster Variation Method R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957). WPI, Tohoku University, Sendai

6. Applications of Belief Propagations • Image Processing K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002). • Low Density Parity Check Codes Y. Kabashima and D. Saad: Statistical mechanics oflow-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004). • CDMA Multiuser Detection Algorithm Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003). T. Tanaka and M. Okada: Approximate belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005). • SatisfabilityProblem O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265(2001). M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002). WPI, Tohoku University, Sendai

7. Interpretation of Belief Propagation based on Information Theory Trial KL Divergence Bethe Free Energy Entropy One-Body Distribution Two-Body Distribution Marginal Probability WPI, Tohoku University, Sendai

8. Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation Prior Probability of natural images is assumed to be the following pairwise Markov random fields: 8 WPI, Tohoku University, Sendai

9. Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation Supervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields: Histogram from Supervised Data M. Yasuda, S. Kataoka and K.Tanaka, J. Phys. Soc. Jpn, Vol.81, No.4, Article No.044801, 2012. 9 WPI, Tohoku University, Sendai

10. Supervised Learning of Pairwise Markov Random Fields by Loopy Belief Propagation Supervised Learning Scheme by Loopy Belief Propagation in Pairwise Markov random fields: 10 WPI, Tohoku University, Sendai

11. Bayesian Image Modeling by Generalized Sparse Prior Assumption: Prior Probability is given as the following Gibbs distribution with the interaction a between every nearest neighbour pair of pixels: p=0: q-state Potts model p=2: Discrete Gaussian Graphical Model 11 WPI, Tohoku University, Sendai

12. Bayesian Image Modeling by Generalized Sparse Prior q=16 p=0.2 Loopy Belief Propagation WPI, Tohoku University, Sendai

13. Bayesian Image Modeling by Generalized Sparse Prior: Conditional Maximization of Entropy Lagrange Multiplier • K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012. Loopy Belief Propagation 13 WPI, Tohoku University, Sendai

14. Bayesian Image Modeling by Generalized Sparse Prior:Conditional Maximization of Entropy Critical Point by Linear Response in LBP q=16 p=0.2 Mandrill Lena • K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012. WPI, Tohoku University, Sendai

15. Prior Analysis by LBP and Conditional Maximization of Entropy in Generalized Sparse Prior Prior Probability q=16 p=0.2 q=16 p=0.5 K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012. 15 WPI, Tohoku University, Sendai

16. Prior Analysis by LBP and Conditional Maximization of Entropy in Generalized Sparse Prior Free Energy of Prior Log-Likelihood for p when the original image f*is given LBP q=16 • K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012. 16 WPI, Tohoku University, Sendai

17. Noise Reductions by Generalized Sparse Prior Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise. WPI, Tohoku University, Sendai 17

18. Noise Reductions by Generalized Sparse Prior Posterior Probability Lagrange Multipliers • K. Tanaka, M. Yasuda and D. M. Titterington: J. Phys. Soc. Jpn, 81, 114802, 2012. 18 WPI, Tohoku University, Sendai

19. Noise Reductions by Generalized Sparse Priors and Loopy Belief Propagation Original Image Degraded Image Restored Image • K. Tanaka, M. Yasuda and • D. M. Titterington: • J. Phys. Soc. Jpn, 81, 114802, 2012. p=0.2 p=0.5 p=1 WPI, Tohoku University, Sendai

20. Noise Reductions by Generalized Sparse Priors and Loopy Belief Propagation Original Image Degraded Image Restored Image • K. Tanaka, M. Yasuda and • D. M. Titterington: • J. Phys. Soc. Jpn, 81, 114802, 2012. p=0.2 p=0.5 p=1 WPI, Tohoku University, Sendai

21. Summary • Formulation of Bayesian image modeling for image processing by means of generalized sparse priors and loopy belief propagation are proposed. • Our formulation is based on the conditional maximization of entropy with some constraints. • In our sparse priors, although the first order phase transitions often appear, our algorithm works well also in such cases. WPI, Tohoku University, Sendai

22. References • S. Kataoka, M. Yasuda, K. Tanaka and D. M. Titterington: Statistical Analysis of the Expectation-Maximization Algorithm with Loopy Belief Propagation in Bayesian Image Modeling, Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.50-63,2012. • M. Yasuda and K. Tanaka: TAP Equation for Nonnegative Boltzmann Machine: Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.192-209, 2012. • S. Kataoka, M. Yasuda and K. Tanaka: Statistical Analysis of Gaussian Image Inpainting Problems, Journal of the Physical Society of Japan, Vol.81, No.2, Article No.025001, 2012. • M. Yasuda, S. Kataoka and K.Tanaka: Inverse Problem in Pairwise Markov Random Fields using Loopy Belief Propagation, Journal of the Physical Society of Japan, Vol.81, No.4, Article No.044801, pp.1-8, 2012. • M. Yasuda, Y. Kabashima and K. Tanaka: Replica Plefka Expansion of Ising systems, Journal of Statistical Mechanics: Theory and Experiment, Vol.2012, No.4, Article No.P04002, pp.1-16, 2012. • K. Tanaka, M. Yasuda and D. M. Titterington: Bayesian image modeling by means of generalized sparse prior and loopy belief propagation, Journal of the Physical Society of Japan, Vol.81, No.11, Article No.114802, 2012. • M. Yasuda and K. Tanaka: Susceptibility Propagation by Using Diagonal Consistency, Physical Review E, Vol.87, No.1, Article No.012134, 2013. WPI, Tohoku University, Sendai