Probabilistic Information Processing in Physical Fluctuomatics

1. Physical FluctuomaticsApplied Stochastic Process1st Review of probabilistic information processing Kazuyuki Tanaka Graduate School of Information Sciences kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Webpage: http://www.smapip.is.tohoku.ac.jp/~kazu/PhysicalFluctuomatics/2010/ Physical Fuctuomatics (Tohoku University)

2. Textbooks • Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) . • Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese). Physical Fuctuomatics (Tohoku University)

3. References of the present lecture • K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002. • Y. Kabashima and D. Saad: Statistical mechanics oflow-density parity-check codes (Topical Review), J. Phys. A, vol.37, no.6, pp.R1-R43, 2004. • H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. • M. Opper and D. Saad D (eds): Advanced Mean Field Methods --- Theory and Practice, MIT Press, 2001. • C. M. Bishop: Pattern Recognition and Machine Learning, Springer, 2006. • M. J. Wainwright and M. I. Jordan: Graphical Models, Exponential Families, and Variational Inference, now Publishing Inc, 2008. • M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2009. Physical Fuctuomatics (Tohoku University)

4. Benefit of Information & Communications Technology • Ubiquitous Computing • Ubiquitous Internet Benefit of Information & Communications Technology Demand for Intelligence It cannot be satisfied only with it being only cheap and being quick. Physical Fuctuomatics (Tohoku University)

5. Information processing according to theories • Inference from propositions • Realization by progress of computational processing capacity • Information processing in real world • Diversity of reason in phenomenon • Compete data is not necessarily obtained. • It is difficult to extract and select some important information from a lot of data. • Uncertainty caused by the gap of knowing simply and understanding actually. • We hope to deal successfully with such uncertainty. Field of Information Processing • Information processing for numerical calculations • Definite Procedure has been given for each calculation. Physical Fuctuomatics (Tohoku University)

6. Computer for next generations Required Capacity • Capability to sympathize with a user （Knowledge) • Capability to put failure and experience to account in the next chance （Learning） How should we deal successfully with the uncertainty caused by the gap of knowing simply and understanding actually? Formulation of knowledge and uncertainty Realization of information processing data with uncertainty Physical Fuctuomatics (Tohoku University)

7. Mathematical expression of uncertainty =>Probability and Statistics Computational model for information processing in data with uncertainty Probabilistic model with graphical structure （Bayesian network） Probabilistic Inference modeling Medical diagnosis Failure diagnosis Risk Management Node is random variable. Arrow is conditional probability. Inference system for data with uncertainty Graph with cycles Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness. Important aspect Physical Fuctuomatics (Tohoku University)

8. Bayes Formula Computational Model for Probabilistic Information Processing Probabilistic Information Processing Probabilistic Model Algorithm • Monte Carlo Method • Markov Chain Monte Carlo Method • Randomized Algorithm • Genetic Algorithm • Approximate Method • Belief Propagation • Mean Field Method Randomness and Approximation Physical Fuctuomatics (Tohoku University)

9. Probabilistic Image Processing K. Tanaka: J. Phys. A, vol.35, 2002. A. S. Willsky: Proceedings of IEEE, vol.90, 2002. Noise Reduction by Probabilistic Image Processing The elements of such a digital array are called pixels. At each point, the intensity of light is represented as an integer number or a real number in the digital image data. Conventional Filter 202 202 192 190 192 190 219 173 202 120 202 120 218 218 100 110 100 110 Modeling of Probabilistic Image Processing based on Conventional Filters Probabilistic Image Processing Markov Random Filed Model Algorithm Physical Fuctuomatics (Tohoku University)

10. Probabilistic Image Processing K. Tanaka: J. Phys. A, vol.35, 2002. A. S. Willsky: Proceedings of IEEE, vol.90, 2002. • Probabilistic Image Processing Degraded Image (Gaussian Noise） MSE:520 MSE: 2137 Lowpass Filter Wiener Filter Median Filter MSE:860 MSE:767 MSE:1040 Physical Fuctuomatics (Tohoku University)

11. Error Correcting Code Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004. error 010 000001111100000 001001011100001 code decode majority rule 010 0 1 0 Error Correcting Codes Parity Check Code Turbo Code, Low Density Parity Check (LDPC) Code High Performance Decoding Algorithm Physical Fuctuomatics (Tohoku University)

12. Error Correcting Codes and Belief Propagation Hokkaido University GCOE Tutorial (Sapporo）

13. Error Correcting Codes and Belief Propagation Hokkaido University GCOE Tutorial (Sapporo）

14. Error Correcting Codes and Belief Propagation Code Word Hokkaido University GCOE Tutorial (Sapporo）

15. Error Correcting Codes and Belief Propagation Received Word 1 1 0 Binary Symmetric Channel 0 1 0 Code Word Hokkaido University GCOE Tutorial (Sapporo）

16. Error Correcting Codes and Belief Propagation Hokkaido University GCOE Tutorial (Sapporo）

17. Error Correcting Codes and Belief Propagation Hokkaido University GCOE Tutorial (Sapporo）

18. Error Correcting Codes and Belief Propagation Hokkaido University GCOE Tutorial (Sapporo）

19. Error Correcting Codes and Belief Propagation Fundamental Concept for Turbo Codes and LDPC Codes Hokkaido University GCOE Tutorial (Sapporo）

20. Received Data Probabilistic model for decoding can be expressed in terms of a physical model for spin glass phenomena CDMA Multiuser Detectors in Mobile Phone Communication T. Tanaka, IEEE Trans. on Information Theory, vol.48, 2002 Relationship between mobile phone communication and spin glass theory Spreading Code Sequence Signals of User A Noise Decode Wireless Communication Coded Signals of Other Users Spreading Code Sequence Coded Signals of User A Physical Fuctuomatics (Tohoku University)

21. Artificial Intelligence • Bayesian Network J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988). Practical algorithms by means of belief propagation Probabilistic inference system Physical Fuctuomatics (Tohoku University)

22. Data is constructed from many bits 010011101110101000111110000110000101000000111010101110101010 101101110001 ０,1 Bit Data A sequence is formed by deciding the arrangement of bits. Materials are constructed from a lot of molecules. Molecule Material Molecules have interactions of each other. System of a lot of elements with mutual relationCommon Concept between Information Sciences and Physics Main Interests Information Processing: Data Physics: Material, Natural Phenomena Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing. A lot of elements have mutual relation of each other Physical Fuctuomatics (Tohoku University)

23. Horizon of Computation in Probabilistic Information Processing Compensation of expressing uncertainty using probability and statistics It must be calculated by taking account of both events with high probability and events with low probability. Computational Complexity It is expected to break throw the computational complexity by introducing approximation algorithms. Physical Fuctuomatics (Tohoku University)

24. What is an important point in computational complexity? • How should we treat the calculation of the summation over 2N configuration? If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40. N fold loops Physical Fuctuomatics (Tohoku University)

25. Why is a physical viewpoint effective in probabilistic information processing? Matrials are constructed from a lot of molecules. (1023 molecules exist in 1 mol.) Molecules have intermolecular forces of each other Theoretical physicists always have to treat such multiple summation. Development of Approximate Methods Probabilistic information processing is also usually reduced to multiple summations or integrations. Application of physical approximate methods to probabilistic information processing Physical Fuctuomatics (Tohoku University)

26. Academic Circulation between Physics and Information Sciences Statistical Mechanical Informatics Probabilistic Information Processing Common Concept Academic Circulation Academic Circulation Statistical Sciences Understanding and prediction of properties of materials and natural phenomena Extraction and processing of information in data Information Sciences Physics Physical Fuctuomatics (Tohoku University)

27. Summary of the present lecture • Probabilistic information processing • Examples of probabilistic information processing • Common concept in physics and information sciences • Application of physical modeling and approximations Future Lectures • Fundamental theory of probability and statistics • Linear model • Graphical model • . • . • . Physical Fuctuomatics (Tohoku University)