Local Quadrature Reconstruction On Smooth Manifolds

1. Local Quadrature Reconstruction On Smooth Manifolds M.Tech Thesis Submitted by BhuwanDhingraY8127167 To the Department of Electrical Engineering IIT Kanpur Supervisors – Prof AmitabhaMukerjee, Prof KS Venkatesh

2. Examples • Image sets with a few degrees of freedom • Disk-Shaped Planar Robot – • Each image lies in but has only 2 degrees of freedom • Images sampled from a 2-d manifold • Other Examples: n = 100 1003, m = 2 n = 100 1003, m = 2 n = 76 1013, m = 1

3. Manifolds • An m-dimensional manifoldis a topological space which resembles the Euclidean space near each point • The manifold itself may lie in but it is everywhere locally homeomorphicto • Generally n >> m • Homeomorphism – A continuous mapping with a continuous inverse

4. Latent Space Manifold points in Global Latent Vectors in Cannot find global latent vectors for – Sphere, Torus, Cylinder etc. as these are not homeomorphicto any Euclidean space.

5. Dimensionality Reduction • Linear – Principal Components Analysis (PCA) • Finds linear subspace in direction of maximum data variance

6. Non-Linear Dimensionality Reduction (NLDR) • Kernel PCA (Scholkopf,1999) – • Applies the kernel trick to project the data to a high-dimensional space followed by normal PCA • ISOMAP (Tenenbaum,2000) – • Preserves geodesic distances between points on the manifold • LLE (Saul,2000) – • Points are expressed as a linear combination of their nearest neighbors, and the relationships are preserved in a global low-dimensional embedding

7. Non-Linear Dimensionality Reduction (NLDR)

8. PCA v NLDR • PCA • Works only for data isometric to a hyperplane in • Provides an explicit mapping between the latent space and manifold • NLDR • Works for arbitrary non-linear manifolds • Gives the embedding only for training points and no mapping between the two spaces

9. Out-of-Sample Points • Out-of-Sample Extension – find for new • Out-of-Sample Reconstruction – find for new

10. Out of Sample Extension • (Bengio,2004) – • Cast several popular NLDR methods into a unified framework as special cases of Kernel PCA with data dependent kernels • Nystrom method used to approximate the out-of-sample extension • (Strange,2011) –

11. Out of Sample Reconstruction • Applications - Video Frame Interpolation

12. Out of Sample Reconstruction • Applications - Generating Novel Views of an Object

13. Out of Sample Reconstruction • Applications - Local Planner Reconstruct to see if collision free Robot Motion PlanningTesting if a local path is collision free

14. Out of Sample Reconstruction • Existing Methods – • Linear interpolation: • Find k-nearest neighbors of new point • Minimize • Reconstruction • Equivalent to fitting a hyper-plane through a small neighborhood on the manifold • Least Squares solution for finding optimal weights requires time

15. Out of Sample Reconstruction • Existing Methods – • Locally Smooth Manifold Learning (Dollar,2006): • Learn a Warping function on the manifold which given a point generates its neighbors using a global regression • Computation time of LSML increases as where is the total number of points on manifold

16. Local Quadrature Reconstruction (LQR) • Consider a local patch on -dimensional hypersurface in () • Take to be the origin and the tangent space to be spanned by the first canonical vectors

17. Local Differential Geometric Model • Smoothness of the manifold implies – • For our choice of coordinate system, • Hessian: : Principal Directions – Span the tangent space : Principal Curvatures

18. Principal Directions and Curvatures

19. Local Differential Geometric Model • First unit vectors in chosen to lie along the principal directions • Ignore higher order terms to get Quadrature Embedding

20. Quadrature Embedding of Smooth Manifolds • Generalization to -dimensional Riemannian manifolds in (Tyagi,2012) Tangent Space Components Normal Space Components

21. Quadratic Regression • If tangent vectors are aligned with principal directions: • If not, we need cross-terms: • In general, for robust estimation: • Need points …+ …+

22. Motivation for LQR • Total curvature parameters in normal space above • Require prohibitively large number of points in for regression • Claim: Directions of high data variance exhibit high curvature • LQR extracts only principal components from the normal space

23. LQR

24. Tangent And Normal Space Estimation • As sampling density on the manifold increases tangent space found by PCA approaches true tangent space (Tyagi,2012) Eigenvectors of PCA on -NN of image shown • - Tangent Vectors • - Normal Vectors

25. 2-d Linear Least Squares

26. Quadratic Regression Test Image LQR Linear

27. Rotating Teapot

28. Regularization • Important to avoid over fitting since is not much greater than • Linear Regression: • Quadratic Regression:

29. Free Parameters

30. Number of Normal Components • For setting we use the following rule – • are the eigenvalues of the covariance matrix for PCA • is set to the minimum value such that • is threshold of data variance we want to consider

31. Complexity • -NN search: or • PCA: • Linear Regression: • Quadratic Regression: • Projection:

32. Rotating Teapot Original Images LQR Linear Reconstructions

33. Disk-Shaped Planar Robot • was set with a energy threshold

34. Disk-Shaped Planar Robot Top – Original Images, Middle – LQR, Bottom - Linear

35. Disk Shaped Planar Robot • Of 200 tested images, LQR outperformed Linear in 183 Size of test point proportional to error

36. Some Failures

37. Video Compression • Video Sequences with few degrees of freedom are low-dimensional trajectories in the space of all images • NLDR methods can be used to assign latent vectors to each frame Retain only frames Total frames in latent vectors in NLDR Transmitter: Reconstruct frames latent vectors and frames LQR Receiver:

38. Foreman Video Sequence

39. Frame Interpolation ,

40. Frame Interpolation ,

41. Features of LQR • Advantages – • Finds better reconstructions than linear interpolation by considering second order terms in time • No training phase • Can be applied to any latent space generated by any NLDR algorithm

42. Features of LQR • Limitation – • Number of neighbors increases with the dimensionality of the manifold as • Need exponentially greater total number of points on the manifold • Computation time increases as or • Over fitting due to large number of parameters in regression • Cannot be used for manifolds with high value of Ex: MNIST digits dataset (), Face datasets ()

43. Thank you

44. Appendix

45. Examples • Curves and surfaces - Spiral Circle n = 2m = 1 n = 3m = 1 Swiss-Roll n = 3m = 2

46. Latent Vectors • Manifolds which are also globallyhomeomorphic to can be endowed with an m-dimensional representation called its Latent Vectors • Latent vectors are not unique • Latent space may be known explicitly from function generating data, or can be found using Dimensionality Reduction

47. Definitions

48. Local Quadrature Reconstruction (LQR) • Restricted to a small neighborhood on the manifold like linear interpolation • Fits a differential geometric model to this neighborhood • Better reconstruction than linear interpolation since we retain up to second order terms in the Taylor series expansion

49. LQR

50. Tangent and Normal Space Estimation (PCA) • : -nearest neighbors of • Eigendecomposition: • Estimated Tangent Space: • Estimated Normal Space: