1 / 98

Image Processing

Image Processing. Cosimo Distante Cosimo.distante@cnr.it Lecture 13: Motion. Last Time: Alignment. Homographies Rotational Panoramas RANSAC Global alignment Warping Blending. Today: Motion and Flow. Motion estimation Patch-based motion (optic flow) Regularization and line processes

shaw
Download Presentation

Image Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Processing CosimoDistante Cosimo.distante@cnr.it Lecture 13: Motion

  2. Last Time: Alignment • Homographies • Rotational Panoramas • RANSAC • Global alignment • Warping • Blending

  3. Today: Motion and Flow • Motion estimation • Patch-based motion (optic flow) • Regularization and line processes • Parametric (global) motion • Layered motion models

  4. Why estimate visual motion? • Visual Motion can be annoying • Camera instabilities, jitter • Measure it; remove it (stabilize) • Visual Motion indicates dynamics in the scene • Moving objects, behavior • Track objects and analyze trajectories • Visual Motion reveals spatial layout • Motion parallax Szeliski

  5. Classes of Techniques • Feature-based methods • Extract visual features (corners, textured areas) and track them • Sparse motion fields, but possibly robust tracking • Suitable especially when image motion is large (10s of pixels) • Direct-methods • Directly recover image motion from spatio-temporal image brightness variations • Global motion parameters directly recovered without an intermediate feature motion calculation • Dense motion fields, but more sensitive to appearance variations • Suitable for video and when image motion is small (< 10 pixels) Szeliski

  6. Patch-based motion estimation

  7. Image motion How do we determine correspondences? Assume all change between frames is due to motion: I J

  8. The Brightness Constraint • Brightness Constancy Equation: Or, equivalently, minimize : Linearizing (assuming small (u,v))using Taylor series expansion: Szeliski

  9. Minimizing: In general Hence, Gradient Constraint (or the Optical Flow Constraint) Szeliski

  10. Patch Translation [Lucas-Kanade] Assume a single velocity for all pixels within an image patch Minimizing LHS: sum of the 2x2 outer product of the gradient vector Szeliski

  11. Lavagna

  12. Local Patch Analysis • How certain are the motion estimates? Szeliski

  13. The Aperture Problem and Let • Algorithm: At each pixel compute by solving • Mis singular if all gradient vectors point in the same direction • e.g., along an edge • of course, trivially singular if the summation is over a single pixel or there is no texture • i.e., only normal flow is available (aperture problem) • Corners and textured areas are OK Szeliski

  14. Pollefeys

  15. SSD Surface – Textured area

  16. SSD Surface -- Edge

  17. SSD – homogeneous area

  18. Iterative Refinement • Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation • Warp one image toward the other using the estimated flow field (easier said than done) • Refine estimate by repeating the process Szeliski

  19. estimate update Initial guess: Estimate: Optical Flow: Iterative Estimation x x0 (usingd for displacement here instead of u) Szeliski

  20. estimate update Initial guess: Estimate: x x0 Optical Flow: Iterative Estimation Szeliski

  21. estimate update Initial guess: Estimate: Initial guess: Estimate: x x0 Optical Flow: Iterative Estimation Szeliski

  22. Optical Flow: Iterative Estimation x x0 Szeliski

  23. Optical Flow: Iterative Estimation • Some Implementation Issues: • Warping is not easy (ensure that errors in warping are smaller than the estimate refinement) • Warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration. • Often useful to low-pass filter the images before motion estimation (for better derivative estimation, and linear approximations to image intensity) Szeliski

  24. actual shift estimated shift Optical Flow: Aliasing Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? nearest match is correct (no aliasing) nearest match is incorrect (aliasing) To overcome aliasing: coarse-to-fine estimation. Szeliski

  25. Limits of the gradient method Fails when intensity structure in window is poor Fails when the displacement is large (typical operating range is motion of 1 pixel) Linearization of brightness is suitable only for small displacements • Also, brightness is not strictly constant in images actually less problematic than it appears, since we can pre-filter images to make them look similar Szeliski

  26. Jw refine warp + u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image J image J image I image I Pyramid of image J Pyramid of image I Coarse-to-Fine Estimation Szeliski

  27. Coarse-to-Fine Estimation I J refine J Jw I warp + I J Jw pyramid construction pyramid construction refine warp + J I Jw refine warp + Szeliski

  28. Spatial Coherence Assumption * Neighboring points in the scene typically belong to the same surface and hence typically have similar motions. * Since they also project to nearby points in the image, we expect spatial coherence in image flow. Black

  29. Formalize this Idea Noisy 1D signal: u x Noisy measurementsu(x) Black

  30. Regularization Find the “best fitting” smoothed function v(x) u v(x) x Noisy measurements u(x) Black

  31. Membrane model Find the “best fitting” smoothed function v(x) u v(x) x Black

  32. Membrane model Find the “best fitting” smoothed function v(x) u v(x) x Black

  33. Membrane model Find the “best fitting” smoothed function v(x) u v(x) Black

  34. Spatial smoothness assumption Faithful to the data Regularization u v(x) x Minimize: Black

  35. Discontinuities u v(x) x What about this discontinuity? What is happening here? What can we do? Black

  36. velocity space u2 + + u1 Robust Estimation Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers. • Sources of outliers (multiple motions): • specularities / highlights • jpeg artifacts / interlacing / motion blur • multiple motions (occlusion boundaries, transparency) Black

  37. Occlusion occlusion disocclusion shear Multiple motions within a finite region. Black

  38. Coherent Motion Possibly Gaussian. Black

  39. Multiple Motions Definitely not Gaussian. Black

  40. Weak membrane model u v(x) x Black

  41. Analog line process Penalty function Family of quadratics Black

  42. Analog line process Infimum defines a robust error function. Minima are the same: Black

  43. Robust Regularization u v(x) x Treat large spatial derivatives as outliers. Minimize: Black

  44. Problem: Least-squares estimators penalize deviations between data & model with quadratic error fn (extremely sensitive to outliers) error penalty function influence function Redescending error functions (e.g., Geman-McClure) help to reduce the influence of outlying measurements. error penalty function influence function Robust Estimation Black

  45. Outlier with respect to neighbors. Robust formulation of spatial coherence term Optical flow Black

  46. http://www.youtube.com/watch?v=V4r2HXGA8jw • http://www.youtube.com/watch?v=ysGM3CfBVpU • http://www.youtube.com/watch?feature=player_detailpage&v=U4taMDEozCs

  47. Optimization • Gradient descent • Coarse-to-fine (pyramid) • Deterministic annealing Black

  48. Parametric motion estimation

More Related