1 / 7

Understanding Expectation Maximization Algorithm: A Key to Parameter Estimation

Explore the Expectation Maximization (EM) algorithm introduced in 1977, a mathematical derivation to estimate parameters from incomplete data. Learn to apply EM for Gaussian mixture model, its steps, intuition, and applications in parameter estimation and Hidden Markov Models.

kearneym
Download Presentation

Understanding Expectation Maximization Algorithm: A Key to Parameter Estimation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Expectation Maximization • First introduced in 1977 • Lots of mathematical derivation • Problem : given a set of data (data is incomplete or having missing values). • Goal : assume the set of data come from a underlying distribution, we need to guess the most likely (maximum likelihood) parameters of that model.

  2. Example • Given a set of data points in R2 • Assume underlying distribution is mixture of Gaussians • Goal: estimate the parameters of each gaussian distribution • Ѳ is the parameter, we consider it consists of means and variances, k is the number of Gaussian model.

  3. Steps of EM algorithm(1) • randomly pick values for Ѳk (mean and variance) • for each xn, associate it with a responsibility value r • rn,k - how likely the nth point comes from/belongs to the kth mixture • how to find r? Assume data come fromthese two distribution

  4. Steps of EM algorithm(2) Distribution by Ѳk Probability that we observe xn in the data set provided it comes from kth mixture Distance between xn and center of kth mixture

  5. Steps of EM algorithm(3) • each data point now associate with (rn,1, rn,2,…, rn,k)rn,k – how likely they belong to kth mixture, 0<r<1 • using r, compute weighted mean and variance for each gaussian model • We get new Ѳ, set it as the new parameter and iterate the process (find new r -> new Ѳ -> ……) • Consist of expectation step and maximization step

  6. Ideas and Intuition • given a set of incomplete (observed) data • assume observed data come from a specific model • formulate some parameters for that model, use this to guess the missing value/data (expectation step) • from the missing data and observed data, find the most likely parameters (maximization step) • iterate step 2,3 and converge

  7. Application • Parameter estimation for Gaussian mixture (demo) • Baum-Welsh algorithm used in Hidden Markov Models • Difficulties • How to model the missing data? • How to determine the number of Gaussian mixture. • What model to be used?

More Related