1 / 95

Physical Fluctuomatics 7th~10th Belief propagation

Physical Fluctuomatics 7th~10th Belief propagation. Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/. Textbooks.

noura
Download Presentation

Physical Fluctuomatics 7th~10th Belief propagation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Physical Fluctuomatics7th~10th Belief propagation Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Physics Fluctuomatics (Tohoku University)

  2. Textbooks • Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 8. • Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., October 2009 (in Japanese), Chapters 6-9. Physics Fluctuomatics (Tohoku University)

  3. What is an important point in computational complexity? • How should we treat the calculation of the summation over 2N configuration? If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. N fold loops • Markov Chain Monte Carlo Method • Belief Propagation Method This Talk Physics Fluctuomatics (Tohoku University)

  4. Bayesian Networks Probabilistic Model and Belief Propagation Bayes Formulas Probabilistic Models Probabilistic Information Processing Belief Propagation J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996). Physics Fluctuomatics (Tohoku University)

  5. Mathematical Formulation of Belief Propagation • Similarity of Mathematical Structures between Mean Field Theory and Bepief Propagation Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory andPractice (MIT Press, 2001). • Generalization of Belief Propagation S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Interpretations of Belief Propagation based on Information Geometry S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004). Physics Fluctuomatics (Tohoku University)

  6. Generalized Extensions of Belief Propagation based on Cluster Variation Method • Generalized Belief Propagation J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005). • Key Technology is the cluster variation method in Statistical Physics R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951). T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, 12 (1957). Physics Fluctuomatics (Tohoku University)

  7. Belief Propagation in Statistical Physics In graphical models with tree graphical structures, Bethe approximation is equivalent to Transfer Matrix Method in Statistical Physics and give us exact results for computations of statistical quantities. In Graphical Models with Cycles, Belief Propagation is equivalent to Bethe approximation or Cluster Variation Method. Bethe Approximation Trandfer Matrix Method (Tree Structures) Belief Propagation Cluster Variation Method (Kikuchi Approximation) Generalized Belief Propagation Physics Fluctuomatics (Tohoku University)

  8. Applications of Belief Propagations • Image Processing K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002). A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002). • Low Density Parity Check Codes Y. Kabashima and D. Saad: Statistical mechanics oflow-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004). • CDMA Multiuser Detection Algorithm Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003). T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005). • Satisfability Problem O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265(2001). M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002). Physics Fluctuomatics (Tohoku University)

  9. Strategy of Approximate Algorithm in Probabilistic Information Processing It is very hard to compute marginal probabilities exactly except some tractable cases. • What is the tractable cases in which marginal probabilities can be computed exactly? • Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases? Physics Fluctuomatics (Tohoku University)

  10. Graphical Representations of Tractable Probabilistic Models X X = X B C D A B C D E = A B C D E Physics Fluctuomatics (Tohoku University)

  11. Graphical Representations of Tractable Probabilistic Models A B C D E X B C D E A B Physics Fluctuomatics (Tohoku University)

  12. Graphical Representations of Tractable Probabilistic Models A B C D E X B C D E A B B C D E A B Physics Fluctuomatics (Tohoku University)

  13. Graphical Representations of Tractable Probabilistic Models A B C D E X B C D E A B B C D E A B A B Physics Fluctuomatics (Tohoku University)

  14. Graphical Representations of Tractable Probabilistic Models A B C D E X B C D E A B B C D E A B A B A B C D E Physics Fluctuomatics (Tohoku University)

  15. Graphical Representations of Tractable Probabilistic Models A B C D E Physics Fluctuomatics (Tohoku University)

  16. Graphical Representations of Tractable Probabilistic Models A B C D E X A B C C D E Physics Fluctuomatics (Tohoku University)

  17. Graphical Representations of Tractable Probabilistic Models A B C D E X A B C C D E X A B C C D E Physics Fluctuomatics (Tohoku University)

  18. Graphical Representations of Tractable Probabilistic Models A B C D E X A B C C D E X A B C C D E B C Physics Fluctuomatics (Tohoku University)

  19. Graphical Representations of Tractable Probabilistic Models A B C D E X A B C C D E X A B C C D E B C B C D E Physics Fluctuomatics (Tohoku University)

  20. Graphical Representations of Tractable Probabilistic Models A B C D E Physics Fluctuomatics (Tohoku University)

  21. Graphical Representations of Tractable Probabilistic Models A B C D E A B C D E Physics Fluctuomatics (Tohoku University)

  22. Graphical Representations of Tractable Probabilistic Models A B C D E A B C D E B C D E Physics Fluctuomatics (Tohoku University)

  23. Graphical Representations of Tractable Probabilistic Models A B C D E A B C D E B C D E C D E Physics Fluctuomatics (Tohoku University)

  24. Graphical Representations of Tractable Probabilistic Models A B C D E A B C D E B C D E C D E D E Physics Fluctuomatics (Tohoku University)

  25. Graphical Representations of Tractable Probabilistic Models X X = X X C C D E A B C E E F A D = C E B F Physics Fluctuomatics (Tohoku University)

  26. Graphical Representations of Tractable Probabilistic Models A D C E B F Physics Fluctuomatics (Tohoku University)

  27. Graphical Representations of Tractable Probabilistic Models A D A D C C E E A A B F B F C C Physics Fluctuomatics (Tohoku University)

  28. Graphical Representations of Tractable Probabilistic Models A D A D C C E E C C B F B F B B A D C E B F Physics Fluctuomatics (Tohoku University)

  29. Graphical Representations of Tractable Probabilistic Models A D A D C C E E B F B F A D D C E C E B F F Physics Fluctuomatics (Tohoku University)

  30. Graphical Representations of Tractable Probabilistic Models A D A D C C E E B F B F A D D C E C E B F F D C E F Physics Fluctuomatics (Tohoku University)

  31. Graphical Representations of Tractable Probabilistic Models A D A D C C E E B F B F A D D C E C E B F F D E C E F F Physics Fluctuomatics (Tohoku University)

  32. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A D C E B F Physics Fluctuomatics (Tohoku University)

  33. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A D C E B F A D E ＝ C E E F B Physics Fluctuomatics (Tohoku University)

  34. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A D C E B F A D E ＝ C E E F B D E ＝ C E E F Physics Fluctuomatics (Tohoku University)

  35. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A D C E B F A D E ＝ C E E F B D D E ＝ ＝ C C E E E F F Physics Fluctuomatics (Tohoku University)

  36. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A D C E B F C D E A ＝ C E B E F C A D C A D E ＝ ＝ C C E E B C E F B F Physics Fluctuomatics (Tohoku University)

  37. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages A D D ＝ ＝ C E C E B F F D A D C C E E B F F A Recursion Formulas for Messages C E C E B Physics Fluctuomatics (Tohoku University)

  38. Graphical Representations of Tractable Probabilistic Models A D Graphical Representation of Marginal Probability in terms of Messages D C E A E C E B F C E C E Step 1 F A D E C B F C D E A D C E F B E C B F D D C E Step 2 C E C E E F F A C E A C Step 3 B A C C E B B Physics Fluctuomatics (Tohoku University)

  39. Graphical Representations of Tractable Probabilistic Models Graphical Representation of Marginal Probability in terms of Messages C A D D Step 1 ＝ A D B ＝ ＝ C E C E C E B F F B F Step 2 A D C E B F Step 3 A D A C E ＝ C E B F B Physics Fluctuomatics (Tohoku University)

  40. 3 4 1 2 6 5 Belief Propagation Probabilistic Models with no Cycles Physics Fluctuomatics (Tohoku University)

  41. 1 3 4 2 1 3 1 4 1 2 2 2 6 6 5 5 Belief Propagation Probabilistic Model on Tree Graph Physics Fluctuomatics (Tohoku University)

  42. 3 4 2 1 6 5 Probabilistic Model on Tree Graph Physics Fluctuomatics (Tohoku University)

  43. 3 4 2 1 6 5 Belief Propagation Probabilistic Model on Tree Graph Physics Fluctuomatics (Tohoku University)

  44. Belief Propagation for Probabilistic Model on Tree Graph No Cycles!! Physics Fluctuomatics (Tohoku University)

  45. Belief Propagation for Probabilistic Model on Square Grid Graph E: Set of all the links Physics Fluctuomatics (Tohoku University)

  46. Belief Propagation for Probabilistic Model on Square Grid Graph Physics Fluctuomatics (Tohoku University)

  47. Belief Propagation for Probabilistic Model on Square Grid Graph Physics Fluctuomatics (Tohoku University)

  48. Marginal Probability Physics Fluctuomatics (Tohoku University)

  49. 2 Marginal Probability Physics Fluctuomatics (Tohoku University)

  50. 2 2 Marginal Probability Physics Fluctuomatics (Tohoku University)

More Related