1 / 35

Learning with Missing Data

This text explores the challenges and techniques involved in learning from incomplete data, including hidden variables and missing values. It discusses different approaches to treating missing data and modeling the data missing mechanism. The concepts of Bayesian estimation, gradient ascent, and Expectation Maximization (EM) are also covered.

mpuente
Download Presentation

Learning with Missing Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning with Missing Data Eran Segal Weizmann Institute

  2. Incomplete Data • Hidden variables • Missing values • Challenges • Foundational – is the learning task well defined? • Computational – how can we learn with missing data?

  3. We need to consider the data missing mechanism Treating Missing Data • How Should we treat missing data? • Case I: A coin is tossed on a table, occasionally it drops and measurements are not taken • Sample sequence: H,T,?,?,T,?,H • Treat missing data by ignoring it • Case II: A coin is tossed, but only heads are reported • Sample sequence: H,?,?,?,H,?,H • Treat missing data by filling it with Tails

  4. Modeling Data Missing Mechanism • X = {X1,...,Xn} are random variables • OX = {OX1,...,OXn} are observability variables • Always observed • Y = {Y1,...,Yn} new random variables • Val(Yi) = Val(Yi)  {?} • Yi is a deterministic function of Xi and OX1:

  5. Modeling Missing Data Mechanism Case I(random missing values) Case II(deliberate missing values)     X X OX OX Y Y

  6. Modeling Missing Data Mechanism Case I(random missing values) Case II(deliberate missing values)     X X OX OX Y Y

  7. Treating Missing Data • When can we ignore the missing data mechanism and focus only on the likelihood? • For every Xi, Ind(Xi;OXi) • Missing at Random (MAR) is sufficient • The probability that the value of Xi is missing is independent of its actual value given other observed values • In both cases, the likelihood decomposes

  8. Hidden (Latent) Variables • Attempt to learn a model with hidden variables • In this case, MAR always holds (variable is always missing) • Why should we care about unobserved variables? X1 X2 X3 X1 X2 X3 H Y1 Y2 Y3 Y1 Y2 Y3 17 parameters 59 parameters

  9. ... Xn X2 X1 Hidden (Latent) Variables • Hidden variables also appear in clustering • Naïve Bayes model: • Class variable is hidden • Observed attributes areindependent given the class Hidden Cluster Observed possible missing values

  10. Likelihood for Complete Data Input Data: X Likelihood: Y • Likelihood decomposes by variables • Likelihood decomposes within CPDs

  11. Likelihood for Incomplete Data Input Data: X Likelihood: Y • Likelihood does not decompose by variables • Likelihood does not decompose within CPDs • Computing likelihood per instance requires inference!

  12. Bayesian Estimation Bayesian network for parameter estimation Bayesian network X X … X[2] X[M] X[1] … Y Y[2] Y[M] Y[1] Y|X=0 Y|X=1 Posteriors are not independent

  13. Identifiability H • Likelihood can have multiple global maxima • Example: • We can rename the values of the hidden variable H • If H has two values, likelihood has two global maxima • With many hidden variables, there can be an exponential number of global maxima • Multiple local and global maxima can also occur with missing data (not only hidden variables) Y

  14. Gradient Ascent: • Follow gradient of likelihood w.r.t. to parameters • Add line search and conjugate gradient methods to get fast convergence MLE from Incomplete Data • Nonlinear optimization problem L(D|) 

  15. Expectation Maximization (EM): • Use “current point” to construct alternative function (which is “nice”) • Guaranty: maximum of new function has better score than current point MLE from Incomplete Data • Nonlinear optimization problem L(D|) 

  16. MLE from Incomplete Data • Nonlinear optimization problem L(D|)  • Gradient Ascent and EM • Find local maxima • Require multiple restarts to find approx. to the global maximum • Require computations in each iteration

  17. How do we compute ? Gradient Ascent • Theorem: • Proof:

  18. Gradient Ascent

  19. Gradient Ascent • Requires computation: P(xi,pai|o[m],) for all i, m • Can be done with clique-tree algorithm, since Xi,Pai are in the same clique

  20. Gradient Ascent Summary • Pros • Flexible, can be extended to non table CPDs • Cons • Need to project gradient onto space of legal parameters • For reasonable convergence, need to combine with advanced methods (conjugate gradient, line search)

  21. Expectation Maximization (EM) • Tailored algorithm for optimizing likelihood functions • Intuition • Parameter estimation is easy given complete data • Computing probability of missing data is “easy” (=inference) given parameters • Strategy • Pick a starting point for parameters • “Complete” the data using current parameters • Estimate parameters relative to data completion • Iterate • Procedure guaranteed to improve at each iteration

  22. Expectation Maximization (EM) • Initialize parameters to 0 • Expectation (E-step): • For each data case o[m] and each family X,U compute P(X,U | o[m], i) • Compute the expected sufficient statisticsfor each x,u • Maximization (M-step): • Treat the expected sufficient statistics as observed and set the parameters to the MLE with respect to the ESS

  23. Updated network X Y E-Step(inference) M-Step(reparameterize) Iterate Expectation Maximization (EM) Initial network X Y + Training data

  24. Expectation Maximization (EM) • Formal Guarantees: • L(D:i+1)  L(D:i) • Each iteration improves the likelihood • If i+1=i , then i is a stationary point of L(D:) • Usually, this means a local maximum • Main cost: • Computations of expected counts in E-Step • Requires inference for each instance in training set • Exactly the same as in gradient ascent!

  25. EM – Practical Considerations • Initial parameters • Highly sensitive to starting parameters • Choose randomly • Choose by guessing from another source • Stopping criteria • Small change in data likelihood • Small change in parameters • Avoiding bad local maxima • Multiple restarts • Early pruning of unpromising starting points

  26. MINVOLSET KINKEDTUBE PULMEMBOLUS INTUBATION VENTMACH DISCONNECT PAP SHUNT VENTLUNG VENITUBE PRESS MINOVL FIO2 VENTALV PVSAT ANAPHYLAXIS ARTCO2 EXPCO2 SAO2 TPR INSUFFANESTH HYPOVOLEMIA LVFAILURE CATECHOL LVEDVOLUME STROEVOLUME ERRCAUTER HR ERRBLOWOUTPUT HISTORY CO CVP PCWP HREKG HRSAT HRBP BP EM in Practice – Alarm Network • Alarm network • Data sampled from true network • 20% of data randomly deleted

  27. EM in Practice – Alarm Network Training error Test error

  28. EM in Practice – Alarm Network

  29. Partial Data: Parameter Estimation • Non-linear optimization problem • Methods for learning: EM and Gradient Ascent • Exploit inference for learning • Challenges • Exploration of a complex likelihood/posterior • More missing data  many more local maxima • Cannot represent posterior  must resort to approximations • Inference • Main computational bottleneck for learning • Learning large networks  exact inference is infeasible resort to approximate inference

  30. Structure Learning w. Missing Data • Distinguish two learning problems • Learning structure for a given set of random variables • Introduce new hidden variables • How do we recognize the need for a new variable? • Where do we introduce a newly added hidden variable within G? • Open ended and less understood…

  31. Structure Learning w. Missing Data • Theoretically, there is no problem • Define score, and search for structure that maximizes it • Likelihood term will require gradient ascent or EM • Practically infeasible • Typically we have O(n2) candidates at each search step • Requires EM for evaluating each candidate • Requires inference for each data instance of each candidate • Total running time per search step:O(n2  M #EM iteration  cost of BN inference)

  32. Typical Search Requires EM A B C Add B D A B D C Requires EM Reverse CB Delete BC A B D A B C C D D Requires EM

  33. Structural EM • Basic idea: use expected sufficient statistics to learn structure, not just parameters • Use current network to complete the data using EM • Treat the completed data as “real” to score candidates • Pick the candidate network with the best score • Use the previous completed counts to evaluate networks in the next step • After several steps, compute a new data completion from the current network

  34. Structural EM • Conceptually • Algorithm maintains an actual distribution Q over completed datasets as well as current structure G and parameters G • At each step we do one of the following • Use <G,G> to compute a new completion Q and redefine G as the MLE relative to Q • Evaluate candidate successors G’ relative to Q and pick best • In practice • Maintain Q implicitly as a model <G,G> • Use the model to compute sufficient statistics MQ[x,u] when these are needed to evaluate new structures • Use sufficient statistics to compute MLE estimates of candidate structures

  35. Structural EM Benefits • Many fewer EM runs • Score relative to completed data is decomposable! • Utilize same benefits as structure learning w. complete data • Each candidate network requires few recomputations • Here savings is large since each sufficient statistics computation requires inference • As in EM, we optimize a simpler score • Can show improvements and convergence • An SEM step that improves in D+ space, improves real score

More Related