Classification of unlabeled data:

1. Classification of unlabeled data: Mixture models, K-means algorithm, and Expectation Maximization (EM) Based on slides by Reza Shadmehr that are based on Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer 2006.

2. 6 6 4 4 2 2 0 0 -2 -2 -4 -4 -6 -6 -10 -8 -6 -4 -2 -10 -8 -6 -4 -2 The problem of finding labels for unlabeled data In nature, items often do not come with labels. How can we learn labels? Unlabeled data Labeled data

3. 1500 1250 1000 750 500 250 0.2 0.4 0.6 0.8 1 Example: image segmentation Identify pixels that are white matter, gray matter, or outside of the brain. Outside the brain Gray matter White matter Pixel value (normalized)

4. Mixtures • If our data is not labeled, we can hypothesize that: • There are exactly m classes in the data: • Each class y occurs with a specific frequency: • Examples of class y are governed by a specific distribution: • According to our hypothesis, each example x(i) must have been generated from a specific “mixture” distribution: • We might hypothesize that the distributions are Gaussian: Parameters of the distributions Mixing proportions Normal distribution

5. Mixture densities Suppose there are only three classes. The data generation process will be something like this: Each data point is generated by a single Gaussian Multinomial random variable that represents the class x belongs to If we had a classifier that could sort our data, we would get labels like this. Here, the idea is that if x is assigned to class i, we set the i-th component of vector z equal to one and keep the remaining components of z at zero.

6. Hidden variable Measured variable

7. Mixture densities Our imagined classifier If our data were labeled, we could estimate the mean and variance of each Gaussian and compute the mixing proportion. Number of times a particular class was present Mixing proportion Mean of each class Variance of each class The K-means algorithm begins by making an initial guess about class center: Step 1: Assign each data point to the class that has the closest center. Step 2: Re-estimate the center of each class.

8. 6 6 6 4 4 4 2 2 2 0 0 0 -2 -2 -2 -4 -4 -4 -6 -6 -6 -10 -8 -6 -4 -2 -10 -8 -6 -4 -2 -10 -8 -6 -4 -2 K-means algorithm on an easy problem K-means (first 5 iterations - converged) K-means classification True labels

9. 6 4 2 2 2 0 0 0 -2 -2 -2 -4 -4 -4 -6 -12 -10 -8 -6 -4 -2 -12 -10 -8 -6 -4 -2 -12 -10 -8 -6 -4 -2 K-means algorithm on a harder problem K-means (first 10 iterations-converged) K-means classification True labels

10. The cost function for the K-means algorithm Step 1: assuming that class centers are known, to minimize cost we should classify a point based on the closest center. Step 2: assuming that class memberships are known, the derivative of cost with respect to centers dictates that centers should be in the middle of the class. Number of points in class j

11. Types of classification data Data set is complete. K-means: Data set is incomplete, but we complete it using a specific cost function. EM: data set is incomplete, but we complete it using posterior probabilities (a “soft” class membership).

12. EM algorithm Instead of using a “hard” class assignment, we could use a “soft” assignment that represents the posterior probability of class membership for each data point: This gives us a “soft” class membership. Using this classifier, we can now re-estimate the parameters of our distribution for each class: “soft” class membership “Number of times” a particular class was present (no longer an integer!)

13. EM algorithm: iteration 0 We assume that there are exactly m classes in the data. We begin with a guess about the initial setting of the parameters: We select m random samples from the dataset and assume that these are the mean of each class. We set the variance of each class to the sampled covariance of the whole data. We could set the mixing coefficients to be equal.

14. EM algorithm: iteration k+1 We have a guess about the mixture parameters: The “E” step: Complete the incomplete data with the posterior probabilities. For each data point, we assign a class membership by computing the posterior probability for all m classes: The “M” step: estimate new mixture parameters based on the new class assignments:

15. 6 4 6 6 6 6 4 4 4 4 2 2 2 2 2 0 0 0 0 0 -2 -2 -2 -2 6 -2 -4 -4 -4 -4 4 -4 -6 -6 -6 -6 2 -10 -8 -6 -4 -2 0 -10 -8 -6 -4 -2 0 -10 -8 -6 -4 -2 0 -10 -8 -6 -4 -2 0 -6 0 -10 -8 -6 -4 -2 -2 -4 -6 -10 -8 -6 -4 -2 0 Example: classification with EM (easy problem) Initial guess 5th iteration 10th iteration 15th iteration To plot each distribution, I draw an ellipse centered at the mean with an area that covers the expected location of the 1st, 2nd, and 3rd quartile of the data. Colors indicate probability of belonging to one class or another. True labeled data Posterior probabilities

16. -870 -880 -890 -900 -910 -920 -930 0 10 20 30 40 EM algorithm: log-likelihood increases with each step

17. 2 2 2 0 0 0 -2 -2 -2 -4 -4 -4 -12 -10 -8 -6 -4 -2 0 -12 -10 -8 -6 -4 -2 0 -12 -10 -8 -6 -4 -2 0 2 -850 0 -852 -854 -2 -856 -4 -858 0 20 40 60 80 -12 -10 -8 -6 -4 -2 0 Example: classification with EM (hard problem) Initial guess 30th iteration 60th iteration Posterior probabilities True labeled data iteration

18. Some observations about EM: EM takes many more iterations to reach convergence than K-means. Each cycles requires significantly more computation. It is common to run K-means first to find a reasonable initialization for EM. Covariance matrices can be set to cluster covariance found for K-means. Mixing proportions can be set to the fraction of data assigned to each cluster in K-means.

19. On the “M” step, EM maximizes the log-likelihood of the data Under the assumption that pi is constant

20. EM and maximizing log-likelihood of the data This is the same as the rule that we had come up with “soft” classification. The rules are the same for estimates of class variance and mixture probabilities. Note that these equation for the parameters are not solutions because the posterior probabilities are themselves function of parameters. What we have done with the algorithm is to try and solve for the parameters iteratively: we made a guess about the parameters, computed the posteriors, and then found the parameters that maximized the likelihood of the data. We iterated until we found convergence.

21. EM algorithm: iteration k+1 We have a guess about the mixture parameters: The “E” step: Compute the posterior probabilities for all data points: The “M” step: under the assumption that the posterior probabilities are constant, estimate new parameter values to maximize log-likelihood: