1 / 28

Optical Flow

Optical Flow. Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan). Classes of Techniques. Feature-based methods Extract salient visual features (corners, textured areas) and track them over multiple frames

betsyl
Download Presentation

Optical Flow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Optical Flow Digital PhotographyCSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)

  2. Classes of Techniques • Feature-based methods • Extract salient visual features (corners, textured areas) and track them over multiple frames • Analyze the global pattern of motion vectors of these features • Sparse motion fields, but possibly robust tracking • Suitable especially when image motion is large (10-s of pixels) • Direct-methods • Directly recover image motion from spatio-temporal image brightness variations • Global motion parameters directly recovered without an intermediate feature motion calculation • Dense motion fields, but more sensitive to appearance variations • Suitable for video and when image motion is small (< 10 pixels) Optical Flow

  3. The Brightness Constraint • Brightness Constancy Equation: Or, better still, Minimize : • Linearizing (assuming small (u,v)): Optical Flow

  4. Local Patch Analysis Optical Flow

  5. Patch Translation [Lucas-Kanade] Assume a single velocity for all pixels within an image patch Minimizing LHS: sum of the 2x2 outer product tensor of the gradient vector Optical Flow

  6. The Aperture Problem and Let • Algorithm: At each pixel compute by solving • Mis singular if all gradient vectors point in the same direction • e.g., along an edge • of course, trivially singular if the summation is over a single pixel or there is no texture • i.e., only normal flow is available (aperture problem) • Corners and textured areas are OK Optical Flow

  7. Aperture Problem and Normal Flow Optical Flow

  8. Local Patch Analysis Optical Flow

  9. Iterative Refinement • Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation • Warp one image toward the other using the estimated flow field (easier said than done) • Refine estimate by repeating the process Optical Flow

  10. Limits of the gradient method Fails when intensity structure in window is poor Fails when the displacement is large (typical operating range is motion of 1 pixel) Linearization of brightness is suitable only for small displacements • Also, brightness is not strictly constant in images actually less problematic than it appears, since we can pre-filter images to make them look similar Optical Flow

  11. Pyramids • Pyramids were introduced as a multi-resolution image computation paradigm in the early 80s. • The most popular pyramid is the Burt pyramid, which foreshadows wavelets • Two kinds of pyramids: • Low pass or “Gaussian pyramid” • Band-pass or “Laplacian pyramid” Optical Flow

  12. Gaussian Pyramid • Convolve image with a small Gaussian kernel • Typically 5x5 • Subsample (decimate by 2) to get lower resolution image • Repeat for more levels • A sequence of low-pass filtered images Optical Flow

  13. Laplacian pyramid • Laplacian as Difference of Gaussian • Band-pass filtered images • Highlights edges at different spatial scales • For matching, this is less sensitive to image illumination changes • But more noisy than using Gaussians Optical Flow

  14. Optical Flow

  15. Jw refine warp + u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image J image J image I image I Pyramid of image J Pyramid of image I Coarse-to-Fine Estimation Optical Flow

  16. Coarse-to-Fine Estimation I J refine J Jw I warp + I J Jw pyramid construction pyramid construction refine warp + J I Jw refine warp + Optical Flow

  17. Global Motion Models • 2D Models: • Affine • Quadratic • Planar projective transform (Homography) • 3D Models: • Instantaneous camera motion models • Homography+epipole • Plane+Parallax Optical Flow

  18. Least Square Minimization (over all pixels): Example: Affine Motion • Substituting into the B.C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns Optical Flow

  19. Quadratic– instantaneous approximation to planar motion Projective – exact planar motion Other 2D Motion Models Optical Flow

  20. Correlation and SSD • For larger displacements, do template matching • Define a small area around a pixel as the template • Match the template against each pixel within a search area in next image. • Use a match measure such as correlation, normalized correlation, or sum-of-squares difference • Choose the maximum (or minimum) as the match • Sub-pixel interpolation also possible Optical Flow

  21. SSD Surface – Textured area Optical Flow

  22. SSD Surface -- Edge Optical Flow

  23. SSD – homogeneous area Optical Flow

  24. Discrete Search vs. Gradient Based Estimation Consider image I translated by The discrete search method simply searches for the best estimate. The gradient method linearizes the intensity function and solves for the estimate Optical Flow

  25. Correlation Window Size • Small windows lead to more false matches • Large windows are better this way, but… • Neighboring flow vectors will be more correlated (since the template windows have more in common) • Flow resolution also lower (same reason) • More expensive to compute • Another way to look at this: • Small windows are good for local search but more precise and less smooth • Large windows good for global search but less precise and more smooth method Optical Flow

  26. Robust Estimation Standard Least Squares Estimation allows too much influence for outlying points Optical Flow

  27. Robust Estimation Robust gradient constraint Robust SSD Optical Flow

  28. References • J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In ECCV’92, pp. 237–252, Italy, May 1992. • M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Understanding, 63(1):75–104, 1996. • H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Trans. Patt. Anal. Mach. Intel., 18(8):814–830, Aug. 1996. • Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In CVPR’97, pp. 520–526, June 1997. Optical Flow

More Related