Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

1. Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image EnsemblesAswin C. Sankaranarayanan Rice University Richard G. BaraniukChinmayHegdeSriramNagaraj

2. Sensor Data Deluge

3. Concise Models • Efficient processing / compression requires concise representation • Our interest in this talk: Collections of images

4. Concise Models • Our interest in this talk: Collections of image parameterized by q\inQ • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw

5. Concise Models • Our interest in this talk: Collections of image parameterized by q\inQ • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw • Image articulation manifold

6. Image Articulation Manifold • N-pixel images: • K-dimensional articulation space • Thenis a K-dimensional manifoldin the ambient space • Very concise model articulation parameter space

7. Smooth IAMs • N-pixel images: • Local isometryimage distance parameter space distance • Linear tangent spacesare close approximationlocally • Low dimensional articulation space articulation parameter space

8. Smooth IAMs • N-pixel images: • Local isometryimage distance parameter space distance • Linear tangent spacesare close approximationlocally • Low dimensional articulation space articulation parameter space

9. Smooth IAMs • N-pixel images: • Local isometryimage distance parameter space distance • Linear tangent spacesare close approximationlocally • Lowdimensional articulation space articulation parameter space

10. Ex: Manifold Learning LLE ISOMAP LE HE Diff. Geo… • K=1rotation

11. Ex: Manifold Learning • K=2rotation and scale

12. Theory/Practice Disconnect Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes] Tangent approximations ?

13. Theory/Practice Disconnect Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donohoand Grimes] Tangent approximations ?

14. Failure of Tangent Plane Approx. • Ex: cross-fading when synthesizing / interpolating images that should lie on manifold Input Image Input Image Linear path Geodesic

15. Theory/Practice Disconnect Isometry • Ex: translation manifold all blue images are equidistant from the red image • Local isometry • satisfied only when sampling is dense

16. Theory/Practice DisconnectNuisance articulations • Unsupervised data, invariably, has additional undesired articulations • Illumination • Background clutter, occlusions, … • Image ensemble is no longer low-dimensional

17. Image representations • Conventional representation for an image • A vector of pixels • Inadequate! pixel image

18. Image representations • Conventional representation for an image • A vector of pixels • Inadequate! • Remainder of the talk • TWOnovel image representations that alleviate the theoretical/practical challenges in manifold learning on image ensembles

19. Transport operators for image manifolds

20. The concept ofTransport operators Beyond point cloud model for image manifolds Example

21. Example: Translation • 2D Translation manifold • Set of all transport operators = • Beyond a point cloud model • Action of the articulation is more accurate and meaningful

22. Optical Flow • Generalizing this idea: Pixel correspondances • Idea: OF between two images is a natural and accurate transport operator OF from I1 to I2 I1 and I2 (Figures from Ce Liu’s optical flow page)

23. Optical Flow Transport • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation IAM Articulations

24. Optical Flow Transport • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation IAM Articulations

25. Optical Flow Transport OFM at • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation • Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth) for large class of articulations IAM Articulations

26. OFM is Smooth (Rotation) Pixel intensityat 3 points Intensity I(θ) Flow (nearly linear) Op. flow v(θ) Articulation θ in [⁰]

27. Main results OFM at • Local model at each • Each point on the OFM defines a transport operator • Each transport operator maps to one of its neighbors • For a large class of articulations, OFMs are smooth and locally isometric • Traditional manifold processing techniques work on OFMs IAM Articulations

28. Linking it all together OFM at Nonlinear dim. reduction IAM The non-differentiability does not dissappear --- it is embedded in the mapping from OFM to the IAM. However, this is a known map Articulations

29. The Story So Far… OFM at Tangent space at IAM IAM Articulations Articulations

30. Input Image Input Image IAM Linear path Geodesic OFM

31. OFM Manifold Learning Data 196 images of two bears moving linearlyand independently Task Find low-dimensional embedding IAM OFM

32. OFM ML + Parameter Estimation Data 196 images of a cup moving on a plane Task 1 Find low-dimensional embedding Task 2 Parameter estimation for new images(tracing an “R”) OFM IAM

33. Karcher Mean • Point on the manifold such that the sum of geodesic distances to every other point is minimized • Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al] 10 images from an IAM ground truth KM linear KM OFM KM

34. Sparse keypoint-based image representation

35. Image representations • Conventional representation for an image • A vector of pixels • Inadequate! pixel image

36. Image representations • Replace vector of pixels with an abstract bagof features • Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptorsfor each keypoint

37. Image representations • Replace vector of pixels with an abstract bagof features • Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptorsfor each keypoint • Feature descriptors are local; it is very easy to make them robust to nuisance imaging parameters

38. Loss of Geometrical Info • Bag of features representations hide potentially useful image geometry Image space Keypoint space • Goal: make salient image geometrical info more explicit for exploitation

39. Key idea • Keypoint space can be endowed with a rich low-dimensional structure in many situations • Mechanism: define kernels ,between keypoint locations, keypoint descriptors

40. Keypoint Kernel • Keypoint space can be endowed with a rich low-dimensional structure in many situations • Mechanism: define kernels ,between keypoint locations, keypoint descriptors • Joint keypoint kernel between two images is given by

41. Keypoint Geometry Theorem: Under the metric induced by the kernel certain ensembles of articulating images formsmooth, isometric manifolds • In contrast: conventional approach to image fusion via image articulation manifolds (IAMs) fraught with non-differentiability (due to sharp image edges) • not smooth • not isometric

42. Application: Manifold Learning • 2D Translation

43. Application: Manifold Learning • 2D Translation • IAM KAM

44. Manifold Learning in the Wild • Rice University’s Duncan Hall Lobby • 158 images • 360° panorama using handheld camera • Varying brightness, clutter

45. Manifold Learning in the Wild • Duncan Hall Lobby • Ground truth using state of the art structure-from-motion software Ground truth IAM KAM

46. Internet scale imagery • Notre-dame cathedral • 738 images • Collected from Flickr • Large variations in illumination (night/day/saturations), clutter (people, decorations),camera parameters (focal length, fov, …) • Non-uniform sampling of the space

47. Organization • k-nearest neighbors

48. Organization • “geodesics’ “zoom-out” “Walk-closer” 3D rotation

49. Summary • Need for novel image representations • Transport operators • Enables differentiability and meaningful transport • Sparse features • Robustness to outliers, nuisance articulations, etc. • Learning in the wild: unsupervised imagery • True power of manifold signal processing lies in fast algorithms that mainly use neighbor relationships • What are compelling applications where such methods can achieve state of the art performance ?