Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009

1. A Global Geometric Framework for Nonlinear Dimensionality ReductionJoshua B. Tenenbaum (Stanford), Vin de Silva (Stanford), John C. Langford (CMU) SCIENCE 2000 Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009

2. Outline • Motivations • PCA, Principle Component Analysis • MDS, Multidimensional Scaling • Isomap, Isometric Feature Mapping • Examples • Summary

3. Motivations • Finding meaningful low-dimensional structures hidden in their high-dimensional observations High-dimensional sensory input, Human Brain, a small number of perceptually relevant features High-dimensional data, Machine Learning, relevant low dimensional features • Discovering the nonlinear degrees of freedom underling complex natural observations: human handwriting, face images, Swiss roll ….

4. 3 degrees of freedom: Up-down pose, Left-right pose, Lighting direction 2 degrees of freedom on the manifold 2 degrees of freedom: Top arch articulation, Bottom loop articulation

5. PCA, Principle Component Analysis PCA finds a low-dimensional embedding of the data points that best preserves their variance as measured in the high dimensional input space Piis the eigenvector corresponding to the ith largest eigenvalue of the covariance matrix of X (Wiki)

6. MDS, Multidimensional Scaling MDS finds an embedding that preserves the pairwise distances (or generalized disparities) between data points, equivalent to PCA when those distances are Euclidean. “An MDS algorithm starts with a matrix of item–item similarities, then assigns a location to each item in N-dimensional space, where N is specified a priori. For sufficiently small N, the resulting locations may be displayed in a graph or 3D visulization.” Wiki Objective: choose to minimize the stress, where is the Euclidian distance between points i and j on the map, and is the dissimilarity between them.

7. Advantages and Limitations of PCA and MDS Advantages • Computational efficiency • Global optimality • Asymptotic convergence guarantees Limitations • Many data sets contain essential nonlinear structures that are invisible to PCA and MDS. (The failure of the Euclidean structure in the input space)

8. Isomap, Isometric Feature MappingBasic idea: • Isomap builds on classical MDS. • It seeks to preserve the intrinsic geometry of the data, as captured in the geodesic manifold distances between all pairs of data points. • The keypoint is estimating the geodesic distance between faraway points, given only input space distances.

9. Isomap: Algorithms • Step 1: Based on the distance in the input space (fixed radius or K nearest neighbors), determining which points are neighbors on the ManifoldM. These neighborhood relations are represented as a weighted graph G over the data points.

10. Isomap: Algorithms • Step 2: Estimating the geodesic distances between all pairs of points on the manifold M by computing their shortest path distances in the graph G. A simple algorithm:

11. Isomap: Algorithms • Step 3: Applying classical MDS to the matrix of graph distances , constructing an embedding of the data in a d-dimensional Euclidean space Y that best preserves the manifold’s estimated intrinsic geometry. Minimizing

12. Isomap: Examples

16. Performance comparison PCA, MDS PCA, MDS Isomap Isomap A. Face images varying in pose and illumination B. Swiss roll data

17. Summary • Isomap is capable of discovering the nonlinear degrees of freedom that underline complex natural observations. • It efficiently (noniterative, polynomial time) computes a globally optimal solution. • It is guaranteed to converge asymptotically to the true structure for intrinsically Euclidean manifolds.