A gentle introduction to Gaussian distribution

1. A gentle introduction to Gaussian distribution

2. X = 0 X = 1 X: Random variable Review • Random variable • Coin flip experiment

3. P(x) P(x) >= 0 0 1 x Review • Probability mass function (discrete) Any other constraints? Hint: What is the sum? Example: Coin flip experiment

4. f(x) f(x) >= 0 x Review • Probability density function (continuous) Unlike discrete, Density function does not represent probability but its rate of change called the “likelihood” Examples?

5. f(x) f(x) >= 0 x Review • Probability density function (continuous) & Integrates to 1.0 x0 X0+dx P( x0 < x < x0+dx ) = f(x0).dx But, P( x = x0 ) = 0

6. The Gaussian Distribution Courtesy: http://research.microsoft.com/~cmbishop/PRML/index.htm

7. A 2D Gaussian

8. Central Limit Theorem The distribution of the sum of N i.i.d. random variables becomes increasingly Gaussian as N grows. Example: N uniform [0,1] random variables.

9. Central Limit Theorem (Coin flip) • Flip coin N times • Each outcome has an associated random variable Xi (=1, if heads, otherwise 0) • Number of heads • NH is a random variable • Sum of N i.i.d. random variables NH = x1 + x2 + …. + xN

10. Central Limit Theorem (Coin flip) • Probability mass function of NH • P(Head) = 0.5 (fair coin) N = 5 N = 10 N = 40

11. Geometry of the Multivariate Gaussian

12. Moments of the Multivariate Gaussian (1) thanks to anti-symmetry of z

13. Moments of the Multivariate Gaussian (2)

14. Maximum likelihood • Fit a probability density model p(x | θ) to the data • Estimate θ • Given independent identically distributed (i.i.d.) data X = (x1, x2, …, xN) • Likelihood • Log likelihood • Maximum likelihood: Maximize ln p(X | θ) w.r.t. θ

15. Maximum Likelihood for the Gaussian (1) Given i.i.d. data , the log likelihood function is given by Sufficient statistics

16. Maximum Likelihood for the Gaussian (2) Set the derivative of the log likelihood function to zero, and solve to obtain Similarly

17. Mixtures of Gaussians (1) Single Gaussian Mixture of two Gaussians Old Faithful data set

18. Mixtures of Gaussians (2) Component Mixing coefficient K=3 Combine simple models into a complex model:

19. Mixtures of Gaussians (3)

20. Mixtures of Gaussians (4) Log of a sum; no closed form maximum. Determining parameters ¹, §, and ¼ using maximum log likelihood Solution: use standard, iterative, numeric optimization methods or the expectation maximization algorithm (Chapter 9).

21. Thank you!