William H. Hsu Thursday, 31 May 2007 Laboratory for Knowledge Discovery in Databases

1. Data Sciences Summer InstituteMultimodal Information Access and SynthesisLearning and Reasoning with Graphical Models of Probability for the Identity Uncertainty Problem William H. Hsu Thursday, 31 May 2007 Laboratory for Knowledge Discovery in Databases Kansas State University http://www.kddresearch.org/KSU/CIS/DSSI-MIAS-SRL-20070531.ppt

2. Part 2 of 8: EM, BNT and gR Tutorial Outline • Suggested Reading • Wikipedia, “Expectation Maximization”: http://snipurl.com/1mwbs • Bilmes, 1998: http://snipurl.com/1mwc7 • Expectation-Maximization (EM) Algorithm • More on EM and Bayesian Learning • EM and unsupervised learning • Lab Practice: Bayes Net Toolbox (BNT) for MATLAB, gR • Applications: Unsupervised Learning and Clustering • Definitions and framework • Constructive induction • Feature construction • Cluster definition • EM, AutoClass, Principal Components Analysis, Self-Organizing Maps

3. Review: BayesOptimal Classification • General-Case BBN Structure Learning: Use Inference to Compute Scores • Optimal Strategy: Bayesian Model Averaging • Assumption: models hH are mutually exclusive and exhaustive • Combine predictions of models in proportion to marginal likelihood • Compute conditional probability of hypothesis h given observed data D • i.e., compute expectation over unknown h for unseen cases • Let h structure, parameters   CPTs Posterior Score Marginal Likelihood Prior over Parameters Prior over Structures Likelihood

4. Unsupervised Learning: Objectives x f(x) y • Unsupervised Learning • Given: data set D • Vectors of attribute values (x1, x2, …, xn) • No distinction between input attributes and output attributes (class label) • Return: (synthetic) descriptor y of each x • Clustering: grouping points (x) into inherent regions of mutual similarity • Vector quantization: discretizing continuous space with best labels • Dimensionality reduction: projecting many attributes down to a few • Feature extraction: constructing (few) new attributes from (many) old ones • Intuitive Idea • Want to map independent variables (x) to dependent variables (y = f(x)) • Don’t always know what “dependent variables” (y) are • Need to discover y based on numerical criterion (e.g., distance metric) Supervised Learning Unsupervised Learning x

5. Clustering as Unsupervised Learning • A Mode of Unsupervised Learning • Given: a collection of data points • Goal: discover structure in the data • Organize data into sensible groups (how many here?) • Criteria: convenient and valid organization of the data • NB: not necessarily rules for classifying future data points • Cluster analysis: study of algorithms, methods for discovering this structure • Representing structure: organizing data into clusters (cluster formation) • Describing structure: cluster boundaries, centers (cluster segmentation) • Defining structure: assigning meaningful names to clusters (cluster labeling) • Cluster: Informal and Formal Definitions • Set whose entities are alike and are different from entities in other clusters • Aggregation of points in the instance space such that distance between any two points in the cluster is less than the distance between any point in the cluster and any point not in it

6. Bayesian Learning and Expectation-Maximization • Problem Definition • Given: data (n-tuples) with missing values, aka partially observable (PO) data • Want to fill in ? with expected value • Solution Approaches • Expected = distribution over possible values • Use “best guess” Bayesian model (e.g., BBN) to estimate distribution • Expectation-Maximization (EM) algorithm can be used here • Intuitive Idea • Want to find hMLin PO case (Dunobserved variablesobserved variables) • Estimation step: calculate E[unobserved variables | h], assuming current h • Maximization step: update wijkto maximize E[lgP(D | h)], Dall variables

7. Expectation-Maximization (EM): A Simple Example [1] Coin Flip1 Flip2 Flip3 • Experiment • Two coins: P(Head on Coin 1) = p, P(Head on Coin 2) = q • Experimenter first selects a coin: P(Coin = 1) =  • Chosen coin tossed 3 times (per experimental run) • Observe: D = {(1 H H T), (1 H T T), (2 T H T)} • Want to predict: , p, q • How to model the problem? • Simple Bayesian network • Now, can find most likely values of parameters , p, q given data D • Parameter Estimation • Fully observable case: easy to estimate p, q, and  • Suppose k heads are observed out of n coin flips • Maximum likelihood estimate vMLfor Flipi: p = k/n • Partially observable case • Don’t know which coin the experimenter chose • Observe: D = {(H H T), (H T T), (T H T)}  {(? H H T), (? H T T), (? T H T)} P(Coin = 1) =  P(Flipi = 1 | Coin = 1) = p P(Flipi = 1 | Coin = 2) = q

8. Expectation-Maximization (EM): A Simple Example [2] • Problem • When we knew Coin = 1 or Coin = 2, there was no problem • No known analytical solution to the partially observable problem • i.e., not known how to compute estimates of p, q, and  to get vML • Moreover, not known what the computational complexity is • Solution Approach: Iterative Parameter Estimation • Given: a guess of P(Coin = 1 | x), P(Coin = 2 | x) • Generate “fictional data points”, weighted according to this probability • P(Coin = 1 | x) = P(x | Coin = 1)P(Coin = 1) / P(x) based on our guess of , p, q • Expectation step (the “E” in EM) • Now, can find most likely values of parameters , p, q given “fictional” data • Use gradient descent to update our guess of , p, q • Maximization step (the “M” in EM) • Repeat until termination condition met (e.g., criterion on validation set) • EM Converges to Local Maxima of the Likelihood Function P(D | )

9. Expectation-Maximization (EM): A Simple Example [3] • Expectation Step • Suppose we observed m actual experiments, each n coin flips long • Each experiment corresponds to one choice of coin (~) • Let h denote the number of heads in experiment xi (a single data point) • Q: How did we simulate the “fictional” data points, E[ log P(x | )]? • A: By estimating (for 1 im, i.e., the real data points) • Maximization Step • Q: What are we updating? What objective function are we maximizing? • A: We are updating to maximize where

10. EM Applications • Unsupervised Learning Problem • Objective: estimate a probability distribution with unobserved variables • Use EM to estimate mixture policy (more on this later; see 6.12, Mitchell) • Pattern Recognition Examples • Human-computer intelligent interaction (HCII) • Detecting facial features in emotion recognition • Gesture recognition in virtual environments • Computational medicine [Frey, 1998] • Determining morphology (shapes) of bacteria, viruses in microscopy • Identifying cell structures (e.g., nucleus) and shapes in microscopy • Other image processing • Many other examples (audio, speech, signal processing; motor control; etc.) • Inference Examples • Plan recognition: mapping from (observed) actions to agent’s (hidden) plans • Hidden changes in context: e.g., aviation; computer security; MUDs

11. Unsupervised Learning Using EM [1] • Bayesian Unsupervised Learning • Given: D = {(x1, x2, …, xn)} (vectors of indistingushed attribute values) • Return: set of class labels that has maximum a posteriori (MAP) probability • Intuitive Idea • Bayesian learning: • MDL/BIC (Occam’s Razor): priors P(h) express “cost of coding” each model h • AutoClass • Define mutually exclusive, exhaustive clusters (class labels) y1, y2, …, yJ • Suppose: each yj (1 jJ) contributes to x • Suppose also: yj’s contribution ~ known pdf,e.g., Mixture of Gaussians • Conjugate priors: priors on y of same form as priors on x • When to Use for Clustering • Believe (or can assume): clusters generated by known pdf • Believe (or can assume): clusters combined using finite mixture

12. Unsupervised Learning Using EM [2] • AutoClass Algorithm [Cheeseman et al, 1988] • Based on maximizing P(x | j, yj, J) • j: class (cluster) parameters (e.g., mean and variance) • yj : synthetic classes (can estimate marginal P(yj) any time) • Apply Bayes’s Theorem, use numerical BOC estimation techniques (cf. Gibbs) • Search objectives • Find best J (ideally:integrate out j, yj; really: start with big J, decrease) • Find j, yj: use MAP estimation, then “integrate in the neighborhood” of yMAP • EM: Find MAP Estimate for P(x | j, yj, J) by Iterative Refinement • Advantages over Symbolic (Non-Numerical) Methods • Returns probability distribution over class membership • More robust than “best” yj • Compare: fuzzy set membership (similar but probability-based) • Can deal with continuous as well as discrete data

13. Clustering with EM and AutoClass [3] • AutoClass Resources • Beginning tutorial (AutoClass II): Cheesemanet al. • Project page: http://ic-www.arc.nasa.gov/ic/projects/bayes-group/autoclass/ • Applications • Knowledge discovery in databases (KDD) and data mining • Infrared astronomical satellite (IRAS): spectral atlas (sky survey) • Molecular biology: pre-clustering DNA acceptor, donor sites (mouse, human) • LandSat data from Kansas (30 km2 region, 1024 x 1024 pixels, 7 channels) • Positive findings: see book chapter by Cheeseman and Stutz, online • Other typical applications: see KD Nuggets (http://www.kdnuggets.com) • Other Tutorials • Obtaining source code from project page • AutoClass III: Lisp implementation [Cheeseman, Stutz, Taylor, 1992] • AutoClass C: C implementation [Cheeseman, Stutz, Taylor, 1998] • General intro by Matteuci: http://snipurl.com/1mwba