Physical Fluctuomatics 7th~10th Belief propagation Appendix

1. Physical Fluctuomatics7th~10th Belief propagationAppendix Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Physics Fluctuomatics (Tohoku University)

2. Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5. Textbooks References • H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. • H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011. • M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010. Physics Fluctuomatics (Tohoku University)

3. Probabilistic Model for Ferromagnetic Materials Physics Fluctuomatics (Tohoku University)

4. Probabilistic Model for Ferromagnetic Materials > = > Prior probability prefers to the configuration with the least number of red lines. Physics Fluctuomatics (Tohoku University)

5. More is different in Probabilistic Model for Ferromagnetic Materials Large p Small p Sampling by Markov Chain Monte Carlo method Disordered State Ordered State Critical Point (Large fluctuation) More is different. Physics Fluctuomatics (Tohoku University)

6. Fundamental Probabilistic Models for Magnetic Materials -1 +1 Since h is positive, the probablity of up spin is larger than the one of down spin． h：External Field Average Variance Physics Fluctuomatics (Tohoku University)

7. Fundamental Probabilistic Models for Magnetic Materials -1 -1 +1 +1 J：Interaction -1 +1 +1 -1 Since J is positive, (a1,a2)=(+1,+1) and (-1,-1) have the largest probability． Average Variance Physics Fluctuomatics (Tohoku University)

8. Fundamental Probabilistic Models for Magnetic Materials E：Set of All the neighbouring Pairs of Nodes h h J J Translational Symmetry Problem: Compute Physics Fluctuomatics (Tohoku University)

9. Fundamental Probabilistic Models for Magnetic Materials h h J J Translational Symmetry Problem: Compute Spontaneous Magnetization Physics Fluctuomatics (Tohoku University)

10. h Jm Jm i Jm Jm Mean Field Approximation for Ising Model We assume that the probability for configurations satisfying are large. Physics Fluctuomatics (Tohoku University)

11. Mean Field Approximation for Ising Model We assume that all random variablesaiare independent of each other, approximately. Fixed Point Equation of m Physics Fluctuomatics (Tohoku University)

12. Fixed Point Equation and Iterative Method • Fixed Point Equation Physics Fluctuomatics (Tohoku University)

13. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University)

14. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University)

15. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University)

16. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University)

17. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University)

18. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics (Tohoku University)

19. h Jm Jm i Jm Jm Marginal Probability Distribution in Mean Field Approximation Jm：Mean Field Physics Fluctuomatics (Tohoku University)

20. h l h l l h Advanced Mean Field Method l：Effective Field Bethe Approximation J Fixed Point Equation for l Kikuchi Method (Cluster Variation Meth) Physics Fluctuomatics (Tohoku University)

21. h h J J Average of Ising Model on Square Grid Graph • Mean Field Approximation • Bethe Approximation • Kikuchi Method (Cluster Variation Method) • Exact Solution （L. Onsager，C.N.Yang） Physics Fluctuomatics (Tohoku University)

22. Model Representation in Statistical Physics Gibbs Distribution Partition Function Energy Function Free Energy Physics Fluctuomatics (Tohoku University)

23. Gibbs Distribution and Free Energy • Gibbs Distribution Free Energy Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a) Free Energy Functional of Trial Probability Distribution Q(a) Physics Fluctuomatics (Tohoku University)

24. Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional Normalization Condition Physics Fluctuomatics (Tohoku University)

25. Kullback-Leibler Divergence and Free Energy Physics Fluctuomatics (Tohoku University)

26. Minimization of Kullback-Leibler Divergence between Interpretation of Mean Field Approximation as Information Theory and Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P] Physics Fluctuomatics (Tohoku University)

27. Interpretation of Mean Field Approximation as Information Theory h h J J Translational Symmetry Problem: Compute Magnetization Physics Fluctuomatics (Tohoku University)

28. Kullback-Leibler Divergence in Mean Field Approximation for Ising Model Physics Fluctuomatics (Tohoku University)

29. Minimization of Kullback-Leibler Divergence and Mean Field Equation Set of all the neighbouring nodes of the node i Variation i Fixed Point Equations for {Qi|iV} Physics Fluctuomatics (Tohoku University)

30. Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model Physics Fluctuomatics (Tohoku University)

31. Conventional Mean Field Equation in Ising Model h h J J Translational Symmetry Fixed Point Equation Physics Fluctuomatics (Tohoku University)

32. Interpretation of Bethe Approximation (1) Translational Symmetry h h J J Compute and Physics Fluctuomatics (Tohoku University)

33. Interpretation of Bethe Approximation (2) Free Energy KL Divergence Physics Fluctuomatics (Tohoku University)

34. Free Energy KL Divergence Interpretation of Bethe Approximation (3) Bethe Free Energy Physics Fluctuomatics (Tohoku University)

35. Interpretation of Bethe Approximation (4) Physics Fluctuomatics (Tohoku University)

36. Interpretation of Bethe Approximation (5) Lagrange Multipliers to ensure the constraints Physics Fluctuomatics (Tohoku University)

37. Interpretation of Bethe Approximation (6) • Extremum Condition Physics Fluctuomatics (Tohoku University)

38. Interpretation of Bethe Approximation (7) Extremum Condition Physics Fluctuomatics (Tohoku University)

39. 3 3 8 1 4 2 1 2 4 7 5 5 6 Interpretation of Bethe Approximation (8) Extremum Condition Physics Fluctuomatics (Tohoku University)

40. 3 1 4 2 5 Interpretation of Bethe Approximation (9) Message Update Rule 3 8 1 2 4 7 5 6 Physics Fluctuomatics (Tohoku University)

41. 3 8 3 1 2 4 7 5 6 1 4 2 3 5 1 4 2 5 Interpretation of Bethe Approximation (10) Message Passing Rule of Belief Propagation = It corresponds to Bethe approximation in the statistical mechanics. Physics Fluctuomatics (Tohoku University)

42. Interpretation of Bethe Approximation (11) Translational Symmetry Physics Fluctuomatics (Tohoku University)

43. Summary • Statistical Physics and Information Theory • Probabilistic Model of Ferromagnetism • Mean Field Theory • Gibbs Distribution and Free Energy • Free Energy and Kullback-Leibler Divergence • Interpretation of Mean Field Approximation as Information Theory • Interpretation of Bethe Approximation as Information Theory Physics Fluctuomatics (Tohoku University)