Physical Fluctuomatics 4th Maximum likelihood estimation and EM algorithm

1. Physical Fluctuomatics4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Phisical Fluctuomatics (Tohoku University)

2. Textbooks • Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4. Phisical Fluctuomatics (Tohoku University)

3. Statistical Machine Learning and Model Selection Inference of Probabilistic Model by using Data Model Selection Statistical Machine Learning Complete Data and Incomplete data Maximum Likelihood EM algorithm Extension Implement by Belief Propagation and Markov Chain Monte Carlo method Akaike Information Criteria, Akaike Bayes Information Criteria, etc. Phisical Fluctuomatics (Tohoku University)

4. Maximum Likelihood Estimation Parameter Phisical Fluctuomatics (Tohoku University)

5. 0 1 2 3 4 5 6 7 8 Maximum Likelihood Estimation Parameter Phisical Fluctuomatics (Tohoku University)

6. 0 1 2 3 4 5 6 7 8 Maximum Likelihood Estimation Data Parameter Data Phisical Fluctuomatics (Tohoku University)

7. 0 1 2 3 4 5 6 7 8 Maximum Likelihood Estimation Data Parameter Data Histogram Phisical Fluctuomatics (Tohoku University)

8. 0 1 2 3 4 5 6 7 8 Maximum Likelihood Estimation Data Parameter Data Histogram Phisical Fluctuomatics (Tohoku University)

9. 0 1 2 3 4 5 6 7 8 Maximum Likelihood Estimation Data Parameter Data Histogram Phisical Fluctuomatics (Tohoku University)

10. Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Phisical Fluctuomatics (Tohoku University)

11. Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Phisical Fluctuomatics (Tohoku University)

12. Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Sample Deviation Sample Average Phisical Fluctuomatics (Tohoku University)

13. Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Sample Deviation Sample Average Histogram Phisical Fluctuomatics (Tohoku University)

14. Maximum Likelihood is unknown Data Hyperparameter Marginal Likelihood Extremum Condition Parameter Bayes Formula Phisical Fluctuomatics (Tohoku University)

15. gi fi i i Probabilistic Model for Signal Processing Noise Transmission Source Signal Observable Data Bayes Formula Phisical Fluctuomatics (Tohoku University)

16. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Prior Probability for Source Signal i j E：Set of all the links Image Data One dimensional Data 1 2 3 4 5 = X 2 3 1 2 3 4 X X 4 5 Phisical Fluctuomatics (Tohoku University)

17. Data Generating Process Additive White Gaussian Noise V：Set of all the nodes Phisical Fluctuomatics (Tohoku University)

18. gi fi i i Probabilistic Model for Signal Processing Parameter データ Hyperparameter Posterior Probability Phisical Fluctuomatics (Tohoku University)

19. Maximum Likelihood in Signal Processing Incomplete Data Parameter Data Marginal Likelihood Hyperparameter Extremum Condition Phisical Fluctuomatics (Tohoku University)

20. Maximum Likelihood and EM algorithm Incomplete Data Parameter Data Marginal Likelihood Q function Hyperparameter Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood. Extemum Condition Phisical Fluctuomatics (Tohoku University)

21. Original Signal 200 100 0 0 255 127 Degraded Signal 200 100 0 0 255 127 Estimated Signal 200 100 0 0 255 127 Model Selection in One Dimensional Signal Expectation Maximization (EM) Algorithm 0.04 0.03 α(t) 0.02 0.01 0 α(0)=0.0001, σ(0)=100 Phisical Fluctuomatics (Tohoku University)

22. Model Selection in Noise Reduction Estimate of Original Image Original Image Degraded Image EM algorithm and Belief Propagation α(0)=0.0001 σ(0)=100 Phisical Fluctuomatics (Tohoku University)

23. Summary • Maximum Likelihood and EM algorithm • Statistical Inference by Gaussian Graphical Model Phisical Fluctuomatics (Tohoku University)

24. Practice 4-1 , , , Let us suppose that data {gi |i=0,1,...,N-1} are generated by according to the following probability density function: , where Prove that estimates for average m and variance s2 of the maximum likelihood , are given as Phisical Fluctuomatics (Tohoku University)