1 / 27

Advanced Machine Learning Techniques: Clustering & Dimensionality Reduction

Learn about K-Means, EM, Spectral Clustering, Dimensionality Reduction, and more in this advanced artificial intelligence lecture. Understand unsupervised learning challenges, EM algorithm, Spectral Clustering advantages, and dimensionality reduction techniques.

bertille-marley
Download Presentation

Advanced Machine Learning Techniques: Clustering & Dimensionality Reduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Advanced Artificial Intelligence Lecture 8: Advance machine learning

  2. Outline • Clustering • K-Means • EM • Spectral Clustering • Dimensionality Reduction

  3. The unsupervised learning problem Many data points, no labels

  4. K-Means Many data points, no labels

  5. Choose a fixed number of clusters Choose cluster centers and point-cluster allocations to minimize error can’t do this by exhaustive search, because there are too many possible allocations. Algorithm fix cluster centers; allocate points to closest cluster fix allocation; compute best cluster centers x could be any set of features for which we can compute a distance (careful about scaling) K-Means

  6. K-Means

  7. K-Means * From Marc Pollefeys COMP 256 2003

  8. K-Means • Is an approximation to EM • Model (hypothesis space): Mixture of N Gaussians • Latent variables: Correspondence of data and Gaussians • We notice: • Given the mixture model, it’s easy to calculate the correspondence • Given the correspondence it’s easy to estimate the mixture models

  9. Expectation Maximzation: Idea • Data generated from mixture of Gaussians • Latent variables: Correspondence between Data Items and Gaussians

  10. Generalized K-Means (EM)

  11. Gaussians

  12. ML Fitting Gaussians

  13. E-Step Learning a Gaussian Mixture(with known covariance) M-Step

  14. Expectation Maximization • Converges! • Proof [Neal/Hinton, McLachlan/Krishnan]: • E/M step does not decrease data likelihood • Converges at local minimum or saddle point • But subject to local minima

  15. Practical EM • Number of Clusters unknown • Suffers (badly) from local minima • Algorithm: • Start new cluster center if many points “unexplained” • Kill cluster center that doesn’t contribute • (Use AIC/BIC criterion for all this, if you want to be formal)

  16. Spectral Clustering

  17. Spectral Clustering

  18. Spectral Clustering: Overview Data Similarities Block-Detection

  19. Eigenvectors and Blocks • Block matrices have block eigenvectors: • Near-block matrices have near-block eigenvectors: [Ng et al., NIPS 02] l2= 2 l3= 0 l1= 2 l4= 0 eigensolver l3= -0.02 l1= 2.02 l2= 2.02 l4= -0.02 eigensolver

  20. Spectral Space • Can put items into blocks by eigenvectors: • Resulting clusters independent of row ordering: e1 e2 e1 e2 e1 e2 e1 e2

  21. The Spectral Advantage • The key advantage of spectral clustering is the spectral space representation:

  22. Measuring Affinity Intensity Distance Texture

  23. Scale affects affinity

  24. Dimensionality Reduction

  25. Dimensionality Reduction with PCA

  26. Linear: Principal Components • Fit multivariate Gaussian • Compute eigenvectors of Covariance • Project onto eigenvectors with largest eigenvalues

More Related