Analysis of Contour Motions

1. Neural Information Processing Systems 2006 Analysis of Contour Motions Ce Liu William T. Freeman Edward H. Adelson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

2. Visual Motion Analysis in Computer Vision • Motion analysis is essential in • Video processing • Geometry reconstruction • Object tracking, segmentation and recognition • Graphics applications • Is motion analysis solved? • Do we have good representation for motion analysis? • Is it computationally feasible to infer the representation from the raw video data? • What is a good representation for motion?

3. Seemingly Simple Examples Kanizsa square From real video

4. Output from the State-of-the-Art Optical Flow Algorithm Kanizsa square Optical flow field T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004

5. Output from the State-of-the-Art Optical Flow Algorithm Dancer Optical flow field T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004

6. Optical flow representation: aperture problem Lines Corners Flat regions Spurious junctions Boundary ownership Illusory boundaries

7. We need motion representation beyond pixel level! Optical Flow Representation Lines Corners Flat regions Spurious junctions Boundary ownership Illusory boundaries

8. J. Wang & E. H. Adelson 1994 Achieved with the help of spatial segmentation Y. Weiss & E. H. Adelson 1994 Layer Representation • Video is a composite of layers • Layer segmentation assumes sufficient textures for each layer to represent motion • A true success?

9. J. Wang & E. H. Adelson 1994 Layer representation is good, but the existing layer segmentation algorithms cannot find the right layers for textureless objects Achieved with the help of spatial segmentation Y. Weiss & E. H. Adelson 1994 Layer Representation • Video is a composite of layers • Layer segmentation assumes sufficient textures for each layer to represent motion • A true success?

10. Challenge: Textureless Objects under Occlusion • Corners are not always trustworthy (junctions) • Flat regions do not always move smoothly (discontinuous at illusory boundaries) • How about boundaries? • Easy to detect and track for textureless objects • Able to handle junctions with illusory boundaries

11. Analysis of Contour Motions • Our approach: simultaneous grouping and motion analysis • Multi-level contour representation • Junctions are appropriated handled • Formulate graphical model that favors good contour and motion criteria • Inference using importance sampling • Contribution • An important component in motion analysis toolbox for textureless objects under occlusion

12. Three Levels of Contour Representation • Edgelets: edge particles • Boundary fragments: a chain of edgelets with small curvatures • Contours: a chain of boundary fragments Forming boundary fragments: easy (for textureless objects) Forming contours: hard (the focus of our work)

13. Overview of our system 1. Extract boundary fragments 2. Edgelet tracking with uncertainty. 3. Boundary grouping and illusory boundary 4. Motion estimation based on the grouping

14. (a) (b) (c) (d) Forming Boundary Fragments • Boundary fragments extraction in frame 1 • Steerable filters to obtain edge energy for each orientation band • Spatially trace boundary fragments • Boundary fragments: lines or curves with small curvature • Temporal edgelet tracking with uncertainties • Frame 1: edgelet (x, y, q) • Frame 2: orientation energy of q • A Gaussian pdf is fit with the weight of orientation energy • 1D uncertainty of motion (even for T-junctions)

15. Forming Contours: Boundary Fragments Grouping • Grouping representation: switch variables(attached to every end of the fragments) • Exclusive: one end connects to at most one other end • Reversible: if end (i,ti) connects to (j,tj), then (j,tj) connects to (i,ti) 1 Arbitrarily possible connection A legal contour grouping 0 Reversibility Another legal contour grouping 1 1 0 0

16. Local Spatial-Temporal Cues for Grouping Illusory boundaries corresponding to the groupings (generated by spline interpolation) Motion stimulus

17. Velocity space Local spatial-temporal cues for grouping: (a) Motion similarity The grouping with higher motion similarity is favored KL( ) < KL( ) Motion stimulus

18. Local spatial-temporal cues for grouping: (b) Curve smoothness The grouping with smoother and shorter illusory boundary is favored Motion stimulus

19. Local spatial-temporal cues for grouping: (c) Contrast consistency The grouping with consistent local contrast is favored Motion stimulus

20. The Graphical Model for Grouping • Affinity metric terms • (a) Motion similarity • (b) Curve smoothness • (c) Contrast consistency • The graphical model for grouping affinity reversibility no self-intersection

21. Motion estimation for grouped contours • Gaussian MRF (GMRF) within a boundary fragment • The motions of two end edgelets are similar if they are grouped together • The graphical model of motion: joint Gaussian given the grouping This problem is solved in early work: Y. Weiss, Interpreting images by propagating Bayesian beliefs, NIPS, 1997.

22. Inference • Two-step inference • Grouping (switch variables) • Motion based on grouping (easy, least square) • Grouping: importance sampling to estimate the marginal of the switch variables • Bidirectional proposal density • Toss the sample if self-intersection is detected • Obtain the optimal grouping from the marginal

23. Why bidirectional proposal in sampling?

24. Why bidirectional proposal in sampling? Affinity metric of the switch variable (darker, thicker means larger affinity) b1b2: 0.39 b1b3: 0.01 b1b4: 0.60 b4b1: 0.20 b4b2: 0.05 b4b3: 0.85 b2b1: 0.50 b2b3: 0.45 b2b4: 0.05 b3b1: 0.01 b3b2: 0.45 b3b4: 0.54 b1b2: 0.1750 b1b3: 0.0001 b1b4: 0.1200 Bidirectional proposal Normalized affinity metrics

25. Why bidirectional proposal in sampling? Bidirectional proposal of the switch variable (darker, thicker means larger affinity) b1b2: 0.39 b1b3: 0.01 b1b4: 0.60 b4b1: 0.20 b4b2: 0.05 b4b3: 0.85 b2b1: 0.50 b2b3: 0.45 b2b4: 0.05 b3b1: 0.01 b3b2: 0.45 b3b4: 0.54 b1b2: 0.62 b1b3: 0.00 b1b4: 0.38 Bidirectional proposal (Normalized) Normalized affinity metrics

26. Motion stimulus Example of Sampling Self intersection

27. Motion stimulus Example of Sampling A valid grouping

28. Motion stimulus Example of Sampling More valid groupings

29. Motion stimulus Example of Sampling More valid groupings

30. From Affinity to Marginals Affinity metric of the switch variable (darker, thicker means larger affinity) Motion stimulus

31. From Affinity to Marginals Marginal distribution of the switch variable (darker, thicker means larger affinity) Greedy algorithm to search for the best grouping based on the marginals Motion stimulus

32. Experiments • All the results are generated using the same parameter settings • Running time depends on the number of boundary fragments, varying from ten seconds to a few minutes in MATLAB

33. Two Moving Bars Frame 1

34. Two Moving Bars Frame 2

35. Two Moving Bars Extracted boundary fragments. The green circles are the boundary fragment end points.

36. Two Moving Bars Optical flow from Lucas-Kanade algorithm. The flow vectors are only plotted at the edgelets

37. Two Moving Bars Estimated motion by our system after grouping

38. Two Moving Bars Boundary grouping and illusory boundaries (frame 1). The fragments belonging to the same contour are plotted in one color.

39. Two Moving Bars Boundary grouping and illusory boundaries (frame 2). The fragments belonging to the same contour are plotted in one color.

40. Kanizsa Square

41. Frame 1

42. Frame 2

43. Extracted boundary fragments

44. Optical flow from Lucas-Kanade algorithm

45. Estimated motion by our system, after grouping

46. Boundary grouping and illusory boundaries (frame 1)

47. Boundary grouping and illusory boundaries (frame 2)

48. Dancer

49. Frame 1

50. Frame 2