Go With The Flow

2. Go With The Flow A New Manifold Modeling and Learning Framework for Image EnsemblesRichard G. BaraniukRice University

3. Go With The Flow A New Manifold Modeling and Learning Framework for Image EnsemblesRichard G. BaraniukRice University AswinSankaranarayananChinmayHegdeSriramNagaraj

4. Sensor Data Deluge

5. Sensor Data Deluge

6. Sensor Data Deluge

7. Concise Models largewaveletcoefficients (blue = 0) pixels • Efficient processing / compression requires concise representation • Sparsity of an individual image

8. Concise Models • Efficient processing / compression requires concise representation • Our interest in this talk: Collections of images

9. Concise Models • Our interest in this talk: Collections of imagesparameterized by q2Q • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw

10. Concise Models • Our interest in this talk: Collections of imagesparameterized by q2Q • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw • Image articulation manifold

11. Image Articulation Manifold • In practice: N-pixel images: • In theory: • K-dimensional articulation space • Thenis a K-dimensional manifoldin the ambient space • Very concise model articulation parameter space

12. Smooth IAMs • In practice: N-pixel images: • In theory: • If images are smooththen so is • Local isometry: image distance parameter space distance • Linear tangent spacesare close approximationlocally articulation parameter space

13. Smooth IAMs • In practice: N-pixel images: • In theory: • If images are smooththen so is • Local isometry: image distance parameter space distance • Linear tangent spacesare close approximationlocally articulation parameter space

14. Ex: Manifold Learning LLE ISOMAP HE … • K=1rotation

15. Ex: Manifold Learning • K=2rotation and scale

16. Theory/Practice Disconnect: Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donoho, Grimes] Local isometry Local tangent approximations

17. Theory/Practice Disconnect: Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donoho, Grimes] Local isometry Local tangent approximations

18. Failure of Local Isometry • Ex: translation manifold • Local isometry all blue images are equidistant from the red image

19. Failure of Tangent Plane Approx. • Ex: cross-fading when synthesizing / interpolating images that should lie on manifold Input Image Input Image Linear path Geodesic

20. Image Articulation Manifold Tangent space at • Linear tangent space at is K-dimensional • provides a mechanism to transport along manifold • problem: since manifold is non-differentiable, tangent approximation is poor • Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process IAM Articulations

21. Optical Flow Transport • Vector field f(x, y) = [u(x, y), v(x, y)] that maps pixels of I1 to pixels of I2 (brightness constancy) • Linearized brightness constancy (LBC) I1 I2

22. History of Optical Flow • Dark ages (<1985) • special cases solved • LBC an under-determined set of linear equations • Horn and Schunk (1985) • Regularization term: smoothness prior on the flow • Brox et al (2005) • shows that linearization of brightness constancy isa bad assumption • develops optimization framework to handle BC directly • Brox et al (2010), Black et al (2010), Liu et al (2010) • practical systems with reliable code

23. Optical Flow Transport • Idea: OF between two images is a natural and accurate transport operator two-image sequence I2image predicted from I1via OF OF from I1 to I2 (Figures from Ce Liu’s optical flow page and ASIFT results page)

24. Optical Flow Transport OFM at • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation • Provides a mechanism to transport along manifold IAM Articulations

25. Optical Flow Manifold OFM at • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation • Provides a mechanism to transport along manifold • Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)[N,S,H,B,2010] IAM Articulations

26. Flow Metric • Replace intensity-based image distance • root cause of IAM non-differentiability … with distance along the OF field from I1 to I2 • Enables safe extension of differential geometric tools • ex: vector fields, curvature, parallel transport, … IAM Flow Field Vt curve c

27. OFM is Smooth (Translation) Pixel intensityat 3 points Flow metric between images(globally linear)

28. OFM is Smooth (Rotation) Pixel intensityat 3 points Intensity I(θ) Flow metric between images(nearly linear) Op. flow v(θ) Articulation θ in [⁰]

29. The Story So Far… OFM at Tangent space at IAM IAM Euclidean metric flow metric Articulations Articulations

30. Input Image Input Image IAM Linear path Geodesic OFM

31. OFM Synthesis

32. Manifold Learning 2D rotations ISOMAP embedding error for OFM and IAM Reference image

33. Manifold Learning 2D rotations Embedding of OFM Reference image

34. OFM Manifold Learning Data 196 images of two bears moving linearlyand independently Task Find low-dimensional embedding IAM OFM

35. OFM ML + Parameter Estimation Data 196 images of a cup moving on a plane Task 1 Find low-dimensional embedding Task 2 Parameter estimation for new images(tracing an “R”) OFM IAM

36. Karcher Mean • Point on the manifold such that the sum of geodesic distances to every other point is minimized • Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al] 10 images from an IAM ground truth KM linear KM OFM KM

37. Manifold Charting • Goal: build a generative model for an entire IAM/OFM based on a small number of base images • El CheapoTM algorithm: • choose a reference image randomly • find all images that can be generated from this image by OF • compute Karcher (geodesic) mean of these images • compute OF from Karcher mean image to other images • repeat on the remaining images until no images remain • Exact representation when no occlusions

38. Manifold Charting • Goal: build a generative model for an entire IAM/OFM based on a small number of base images • Ex: cube rotating about axis • All cube images can be representing using 4 reference images + OFMs • Many applications • selection of target templates for classification • “next-view” selection for adaptive sensing applications IAM

39. Summary • IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints • But practical IAMs are non-differentiable • IAM-based algorithms have not lived up to their promise • Optical flow manifolds (OFMs) • smooth even when IAM is not • OFM ~ nonlinear tangent space • support accurate image synthesis, learning, charting, … • Barely discussed here: OF enables the safe extension of differential geometry concepts • Log/Exp maps, Karcher mean, parallel transport, …

40. Related Work • Analytic transport operators • transport operator has group structure [Xiao and Rao07][Culpepper and Olshausen09] [Miller and Younes01] [Tuzel et al 08] • non-linear analytics [Dollar et al 06] • spatio-temporal manifolds [Li and Chellappa10] • shape manifolds [Klassen et al 04] • Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups) • In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations

41. Open Questions • Our treatment is specific to image manifolds • What are the natural transport operators for other data manifolds? dsp.rice.edu

43. Open Questions Theorem: random measurements stably embed aK-dim manifoldwhp[B, Wakin, FOCM’08] Q: Is there an analogousresult for OFMs?