Dimensionality Reduction Part 2: Nonlinear Methods

1. Dimensionality ReductionPart 2: Nonlinear Methods Comp 790-090 Spring 2007

2. Why Dimensionality Reduction • Two approaches to reduce number of features • Feature selection: select the salient features by some criteria • Feature extraction: obtain a reduced set of features by a transformation of all features • Data visualization and exploratory data analysis also need to reduce dimension • Usually reduce to 2D or 3D

3. Deficiencies of Linear Methods • Data may not be best summarized by linear combination of features • Example: PCA cannot discover 1D structure of a helix

4. Intuition: how does your brain store these pictures?

5. Brain Representation

6. Brain Representation • Every pixel? • Or perceptually meaningful structure? • Up-down pose • Left-right pose • Lighting direction So, your brain successfully reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!

7. latent Y X observed Manifold Learning • Discover low dimensional representations (smooth manifold) for data in high dimension. • Linear approaches(PCA, MDS) • Non-linear approaches (ISOMAP, LLE, others)

8. Linear Approach- PCA • PCA Finds subspace linear projections of input data.

9. Linear Method • Linear Methods for Dimensionality Reduction • PCA: rotate data so that principal axes lie in direction of maximum variance • MDS: find coordinates that best preserve pairwise distances PCA

10. Motivation • Linear Dimensionality Reduction doesn’t always work • Data violates underlying “linear”assumptions • Data is not accurately modeled by “affine” combinations of measurements • Structure of data, while apparent, is not simple • In the end, linear methods do nothing more than “globally transform” (rate, translate, and scale) all of the data, sometime what’s needed is to “unwrap” the data first

11. What does PCA Really Model? • Principle Component Analysis assumptions • Mean-centered distribution • What if the mean, itself is atypical? • Eigenvectors ofCovariance • Basis vectors alignedwith successive directionsof greatest variance • Classic 1st Orderstatistical model • Distribution is characterizedby its mean and variance (Gaussian Hyperspheres)

12. Nonlinear Approaches- Isomap Josh. Tenenbaum, Vin de Silva, John langford 2000 • Constructing neighbourhood graph G • For each pair of points in G, Computing shortest path distances ---- geodesic distances. • Use Classical MDS with geodesic distances. Euclidean distance Geodesic distance

13. Sample points with Swiss Roll • Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000.

14. Construct neighborhood graph G K- nearest neighborhood (K=7) DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A)

15. Compute all-points shortest path in G Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold (figure B)

16. Use MDS to embed graph in Rd • Find a d-dimensional Euclidean space Y (Figure c) to preserve the pariwise diatances.

17. Isomap • Key Observation:On a manifold distances are measured using geodesic distances rather than Euclidean distances Small Euclidean distance Large geodesic distance

18. Problem: How to Get Geodesics • Without knowledge of the manifold it is difficult to compute the geodesic distance between points • It is even difficult if you know the manifold • Solution • Use a discrete geodesic approximation • Apply a graph algorithm to approximate the geodesic distances

19. Dijkstra’s Algorithm • Efficient Solution to all-points-shortest path problem • Greedy breath-first algorithm

20. Dijkstra’s Algorithm • Efficient Solution to all-points-shortest path problem • Greedy breath-first algorithm

21. Dijkstra’s Algorithm • Efficient Solution to all-points-shortest path problem • Greedy breath-first algorithm

22. Dijkstra’s Algorithm • Efficient Solution to all-points-shortest path problem • Greedy breath-first algorithm

23. Isomap algorithm • Compute fully-connected neighborhood of points for each item • Can be k nearest neighbors or ε-ball • Neighborhoods must be symmetric • Test that resulting graph is fully-connected, if not increase either K or  • Calculate pairwise Euclidean distances within each neighborhood • Use Dijkstra’s Algorithm to compute shortest path from each point to non-neighboring points • Run MDS on resulting distance matrix

24. Isomap Results • Find a 2D embedding of the 3D S-curve (also shown for LLE) • Isomap does a good job of preserving metric structure (not surprising) • The affine structure is also well preserved

25. Residual Fitting Error

26. Neighborhood Graph

27. More Isomap Results

28. More Isomap Results

29. Isomap Failures • Isomap also has problems on closed manifolds of arbitrary topology

30. Local Linear Embeddings • First Insight • Locally, at a fine enough scale, everything looks linear

31. Local Linear Embeddings • First Insight • Find an affine combination the “neighborhood” about a point that best approximates it

32. Finding a Good Neighborhood • This is the remaining “Art” aspect of nonlinear methods • Common choices • -ball: find all items that lie within an epsilon ball of the target item as measured under some metric • Best if density of items is high and every point has a sufficient number of neighbors • K-nearest neighbors: find the k-closest neighbors to a point under some metric • Guarantees all items are similarly represented, limits dimension to K-1

33. Rn Rn M M z x: coordinate for z R2 R2 x2 x2 x x1 x1 Characterictics of a Manifold Locally it is a linear patch Key: how to combine all local patches together?

34. LLE: Intuition • Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood • Approximation error, e(W), can be made small • Local neighborhood is effected by the constraint Wij=0 if zi is not a neighbor of zj • A good projection should preserve this local geometric property as much as possible

35. LLE: Intuition We expect each data point and its neighbors to lie on or close to a locally linear patch of the manifold. Each point can be written as a linear combination of its neighbors. The weights chosen to minimize the reconstruction Error.

36. LLE: Intuition • The weights that minimize the reconstruction errors are invariant to rotation, rescaling and translation of the data points. • Invariance to translation is enforced by adding the constraint that the weights sum to one. • The weights characterize the intrinsic geometric properties of each neighborhood. • The same weights that reconstruct the data points in D dimensions should reconstruct it in the manifold in d dimensions. • Local geometry is preserved

37. LLE: Intuition Low-dimensional embedding the i-th row of W Use the same weights from the original space

38. Local Linear Embedding (LLE) • Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood • Approximation error, e(W), can be made small • Meaning of W: a linear representation of every data point by its neighbors • This is an intrinsic geometrical property of the manifold • A good projection should preserve this geometric property as much as possible

39. Constrained Least Square problem Compute the optimal weight for each point individually: Neightbors of x Zero for all non-neighbors of x

40. Finding a Map to a Lower Dimensional Space • Yi in Rk: projected vector for Xi • The geometrical property is best preserved if the error below is small • Y is given by the eigenvectors of the lowest d non-zero eigenvalues of the matrix Use the same weights computed above

41. Numerical Issues • Numerical problems can arise in computing LLEs • The least-squared covariance matrix that arises in the computation of the weighting matrix, W, solution can be ill-conditioned • Regularization (rescale the measurements by adding a small multiple of the Identity to covariance matrix) • Finding small singular (eigen) values is not as well conditioned as finding large ones. The small ones are subject to numerical precision errors, and to get mixed • Good (but slow) solvers exist, you have to use them

42. Results • The resulting parameter vector, yi, gives the coordinates associated with the item xi • The dth embedding coordinate is formed from the orthogonal vector associated with thedst singular value of A.

43. Reprojection • Often, for data analysis, a parameterization is enough • For interpolation and compression we might want to map points from the parameter space back to the “original” space • No perfect solution, but a few approximations • Delauney triangulate the points in the embedding space, find the triangle that the desired parameter setting falls into, and compute the baricenric coordinates of it, and use them as weights • Interpolate by using a radially symmetric kernel centered about the desired parameter setting • Works, but mappings might not be one-to-one

44. LLE Example • 3-D S-Curve manifold with points color-coded • Compute a 2-D embedding • The local affine structure is well maintained • The metric structure is okay locally, but can drift slowly over the domain (this causes the manifold to taper)

45. More LLE Examples

46. More LLE Examples

47. LLE Failures • Does not work on to closed manifolds • Cannot recognize Topology

48. Summary • Non-Linear Dimensionality Reduction Methods • These methods are considerably more powerful and temperamental than linear method • Applications of these methods are a hot area of research • Comparisons • LLE is generally faster, but more brittle than Isomaps • Isomaps tends to work better on smaller data sets(i.e. less dense sampling) • Isomaps tends to be less sensitive to noise (perturbation of the input vectors) • Issues • Neither method handles closed manifolds and topological variations well