Linear response formula and generalized belief propagation for probabilistic inference

1. Linear response formula and generalized belief propagation for probabilistic inference Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University, Sendai 980-8579, Japan kazu@statp.is.tohoku.ac.jp http://www.statp.is.tohoku.ac.jp/~kazu/ Reference K. Tanaka: Probabilistic Inference by means of Cluster Variation Method and Linear Response Theory, IEICE Trans. on Inf. & Syst., vol.E86-D, no.7, pp.1228-1242, 2003. SMAPIP-MSI2004

2. Contents Introduction Probabilistic Inference Cluster Variation Method and Generalized Belief Propagation Linear Response Concluding Remarks SMAPIP-MSI2004

3. Introduction Bayes Formula Probabilistic Inference Probabilistic Model Belief Marginal Probability Probabilistic Inference and Belief Propagation Probabilistic models on networks with some loops =>Good Approximation Probabilistic models on tree-like networks with no loops =>Exact Results Belief Propagation Generalization SMAPIP-MSI2004

4. Introduction Probabilistic model with no loop Belief Propagation (Lauritzen, Pearl) Probabilistic model with some loops Probabilistic model with no loop Transfer Matrix Statistical Mechanics and Belief Propagation Transfer Matrix = Belief Propagation Recursion Formula for Beliefs and Messages Bethe/Kikuchi Method Cluster Variation Method Probabilistic model with some loops SMAPIP-MSI2004

5. Review of generalized loopy belief propagation based on cluster variation method. • Calculation of correlations between any pair of nodes by combining the cluster variation method with the linear response theory Purpose SMAPIP-MSI2004

6. Contents Introduction Probabilistic Inference Cluster Variation Method and Generalized Belief Propagation Linear Response Concluding Remarks SMAPIP-MSI2004

7. Probabilistic Inference SMAPIP-MSI2004

8. Probabilistic Inference Probabilistic Inference and Probabilistic Model SMAPIP-MSI2004

9. Probabilistic Inference Probabilistic Inference and Probabilistic Model SMAPIP-MSI2004

10. Probabilistic Inference Probabilistic Model and Beliefs Statistics: Marginal Probability Statistical Mechanics: One-body distribution Probabilistic Inference: Belief SMAPIP-MSI2004

11. Contents Introduction Probabilistic Inference Cluster Variation Method and Generalized Belief Propagation Linear Response Concluding Remarks SMAPIP-MSI2004

12. Belief Propagation • Belief Propagation is equivalent to Bethe Approximation Y. Kabashima and D. Saad: Europhys. Lett. 1998. • Correctness of Loopy Belief Propagation Y. Weiss: Neural Computation 2000. Y. Weiss and W. T. Freeman: Neural Computation 2001. • Generalized Belief Propagation was proposed based on Cluster Variation Method J. S. Yedidia, W. T. Freeman and Y. Weiss: NIPS 2000. • Interpretation of Generalized Belief Propagation based on Information Geometry S. Ikeda, T. Tanaka and S. Amari: Neural Computation 2004. SMAPIP-MSI2004

13. Cluster Variation Method • Original Formulation R. Kikuchi: Phys. Rev. 1951. T. Morita: J. Phys. Soc. Jpn 1957 J. Math. Phys. 1972 J. Stat. Phys. 1990. • Extension to Random Spin System T. Morita: Physica A, 1979. T. Horiguchi: Physica A, 1981. • Convergence of a Sequence of Approximations in CVM A. G. Schlijper: Phys. Rev. B 1983 J. Stat. Phys. 1985. SMAPIP-MSI2004

14. Cluster Variation Method and Generalized Belief Propagation SMAPIP-MSI2004

15. Generalized Loopy Belief Propagation Extreme Condition of Kullback-Leibler Divergence Message Passing Algorithm SMAPIP-MSI2004

16. Generalized Loopy Belief Propagation Expression of Marginal Probability in terms of Messages SMAPIP-MSI2004

17. Generalized Loopy Belief Propagation SMAPIP-MSI2004

18. GBP Numerical Experiments Exact SMAPIP-MSI2004

19. Numerical Experiments SMAPIP-MSI2004

20. Contents Introduction Probabilistic Inference Cluster Variation Method and Generalized Belief Propagation Linear Response Concluding Remarks SMAPIP-MSI2004

21. Linear Response Theory (LRT) • MFA + LRT • Stat. Phys. [R. J. Elliott and W. Marshall: Rev. Mod. Phys. 1958] • [M. E. Fisher and R. J. Bufford: Phys. Rev. 1967] • NIPS [T. Tanaka: Phys. Rev. 1998] • [H. J. Kappen et al Neural Computation 1998] • Bethe Approx. (BP) + LRT • Stat. Phys. [T. Morita: J. Phys. Soc. Jpn. 1968] • NIPS [M. Welling and Y. W. The: Neural Computation 2004] • Kramers-Wannier-Kikuchi Approx. + LRT • Stat. Phys. [J. M. Sanchez: Physica A 1982] • CVM (GBP) + LRT • Stat. Phys. [T. Morita: Prog. Theor. Phys. 1991] • [K. Tanaka, T. Horiguchi and T. Morita 1991] • NIPS [K. Tanaka: IEICE Trans. Inf. & Syst. 2003] Co-variance between any pair of nodes SMAPIP-MSI2004

22. Linear Response SMAPIP-MSI2004

23. Final Result SMAPIP-MSI2004

24. Basic Cluster and Subcluster • Example SMAPIP-MSI2004

25. Probabilistic Model • Joint Probability Distribution Example SMAPIP-MSI2004

26. Cluster Variation Method • Kullback-Leibler Divergence and Kikuchi Free Energy SMAPIP-MSI2004

27. Present Probabilistic Model Marginal Distribution in CVM SMAPIP-MSI2004

28. Probabilistic Model with External Field Marginal Distribution in CVM SMAPIP-MSI2004

29. The approximate marginal probability is expanded in powers of Lagrange multipliers and only the first order terms are remained. Linear Response in CVM SMAPIP-MSI2004

30. This equation can be regarded as a system of linear equations for Lagrange multipliers. Linear Response in CVM SMAPIP-MSI2004

31. Correlation Function in CVM SMAPIP-MSI2004

32. Correlation Function in CVM Example SMAPIP-MSI2004

33. GBP+LR Numerical Experiments Exact SMAPIP-MSI2004

34. GBP Numerical Experiments GBP + Linear Response SMAPIP-MSI2004

35. Contents Introduction Probabilistic Inference Cluster Variation Method and Generalized Belief Propagation Linear Response Concluding Remarks SMAPIP-MSI2004

36. Cluster Variation Method + Linear Response Theory → General CVM Approximate Formula for Correlation Concluding Remarks Future Problems • Accuracy of GBP • Inconsistency between GBP and GBP+LRT • Statistical Learning of Conditional Probability. •  Maximum Likelihood Framework • EM algorithm SMAPIP-MSI2004