1 / 116

Motion estimation

Motion estimation. Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski. Why Visual Motion?. Visual Motion can be annoying Camera instabilities, jitter Measure it. Remove it. Visual Motion indicates dynamics in the scene Moving objects, behavior

nanji
Download Presentation

Motion estimation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Motion estimation Introduction to Computer VisionCS223B, Winter 2005Richard Szeliski

  2. Why Visual Motion? • Visual Motion can be annoying • Camera instabilities, jitter • Measure it. Remove it. • Visual Motion indicates dynamics in the scene • Moving objects, behavior • Track objects and analyze trajectories • Visual Motion reveals spatial layout of the scene • Motion parallax Motion estimation

  3. Today’s lecture • Motion estimation • background: image pyramids, image warping • application: image morphing • parametric motion (review) • optic flow • layered motion models Motion estimation

  4. Image Pyramids

  5. Image Pyramids Motion estimation

  6. Pyramid Creation • “Laplacian” Pyramid • Created from Gaussianpyramid by subtractionLl = Gl – expand(Gl+1) filter mask “Gaussian” Pyramid Motion estimation

  7. Octaves in the Spatial Domain Lowpass Images • Bandpass Images Motion estimation

  8. Pyramids • Advantages of pyramids • Faster than Fourier transform • Avoids “ringing” artifacts • Many applications • small images faster to process • good for multiresolution processing • compression • progressive transmission • Known as “mip-maps” in graphics community • Precursor to wavelets • Wavelets also have these advantages Motion estimation

  9. Laplacian level 4 Laplacian level 2 Laplacian level 0 Motion estimation left pyramid right pyramid blended pyramid

  10. Pyramid Blending Motion estimation

  11. Image Warping

  12. Parametric (global) warping • Examples of parametric warps: aspect rotation translation perspective cylindrical affine Motion estimation

  13. Image Warping • Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’)=f(h(x))? h(x) x x’ f(x) g(x’) Motion estimation

  14. Forward Warping • Send each pixel f(x) to its corresponding location x’=h(x) in g(x’) • What if pixel lands “between” two pixels? h(x) x x’ f(x) g(x’) Motion estimation

  15. Forward Warping • Send each pixel f(x) to its corresponding location x’=h(x) in g(x’) • What if pixel lands “between” two pixels? • Answer: add “contribution” to several pixels, normalize later (splatting) h(x) x x’ f(x) g(x’) Motion estimation

  16. Inverse Warping • Get each pixel g(x’) from its corresponding location x=h-1(x’) in f(x) • What if pixel comes from “between” two pixels? h-1(x’) x x’ f(x) g(x’) Motion estimation

  17. Inverse Warping • Get each pixel g(x’) from its corresponding location x=h-1(x’) in f(x) • What if pixel comes from “between” two pixels? • Answer: resample color value from interpolated (prefiltered) source image x x’ f(x) g(x’) Motion estimation

  18. Interpolation • Possible interpolation filters: • nearest neighbor • bilinear • bicubic (interpolating) • sinc / FIR • Needed to prevent “jaggies” and “texture crawl” (see demo) Motion estimation

  19. Prefiltering • Essential for downsampling (decimation) to prevent aliasing • MIP-mapping [Williams’83]: • build pyramid (but what decimation filter?): • block averaging • Burt & Adelson (5-tap binomial) • 7-tap wavelet-based filter (better) • trilinear interpolation • bilinear within each 2 adjacent levels • linear blend between levels (determined by pixel size) Motion estimation

  20. Prefiltering • Essential for downsampling (decimation) to prevent aliasing • Other possibilities: • summed area tables • elliptically weighted Gaussians (EWA) [Heckbert’86] Motion estimation

  21. Image Warping – non-parametric • Specify more detailed warp function • Examples: • splines • triangles • optical flow (per-pixel motion) Motion estimation

  22. Image Warping – non-parametric • Move control points to specify spline warp Motion estimation

  23. Image Morphing

  24. Image Morphing • How can we in-between two images? • Cross-dissolve(all examples from [Gomes et al.’99]) Motion estimation

  25. Image Morphing • How can we in-between two images? • Warp then cross-dissolve = morph Motion estimation

  26. Warp specification • How can we specify the warp? • Specify corresponding points • interpolate to a complete warping function • Nielson, Scattered Data Modeling, IEEE CG&A’93] Motion estimation

  27. Warp specification • How can we specify the warp? • Specify corresponding vectors • interpolate to a complete warping function Motion estimation

  28. Warp specification • How can we specify the warp? • Specify corresponding vectors • interpolate [Beier & Neely, SIGGRAPH’92] Motion estimation

  29. Warp specification • How can we specify the warp? • Specify corresponding spline control points • interpolate to a complete warping function Motion estimation

  30. Final Morph Result Motion estimation

  31. Motion estimation

  32. Classes of Techniques • Feature-based methods • Extract salient visual features (corners, textured areas) and track them over multiple frames • Analyze the global pattern of motion vectors of these features • Sparse motion fields, but possibly robust tracking • Suitable especially when image motion is large (10-s of pixels) • Direct-methods • Directly recover image motion from spatio-temporal image brightness variations • Global motion parameters directly recovered without an intermediate feature motion calculation • Dense motion fields, but more sensitive to appearance variations • Suitable for video and when image motion is small (< 10 pixels) Motion estimation

  33. The Brightness Constraint • Brightness Constancy Equation: Or, better still, Minimize : • Linearizing (assuming small (u,v)): Motion estimation

  34. Local Patch Analysis Motion estimation

  35. Patch Translation [Lucas-Kanade] Assume a single velocity for all pixels within an image patch Minimizing LHS: sum of the 2x2 outer product tensor of the gradient vector Motion estimation

  36. The Aperture Problem and Let • Algorithm: At each pixel compute by solving • Mis singular if all gradient vectors point in the same direction • e.g., along an edge • of course, trivially singular if the summation is over a single pixel or there is no texture • i.e., only normal flow is available (aperture problem) • Corners and textured areas are OK Motion estimation

  37. Aperture Problem and Normal Flow Motion estimation

  38. Local Patch Analysis Motion estimation

  39. Iterative Refinement • Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation • Warp one image toward the other using the estimated flow field (easier said than done) • Refine estimate by repeating the process Motion estimation

  40. estimate update Initial guess: Estimate: Optical Flow: Iterative Estimation x x0 Motion estimation

  41. estimate update Initial guess: Estimate: x x0 Optical Flow: Iterative Estimation Motion estimation

  42. estimate update Initial guess: Estimate: Initial guess: Estimate: x x0 Optical Flow: Iterative Estimation Motion estimation

  43. Optical Flow: Iterative Estimation x x0 Motion estimation

  44. Optical Flow: Iterative Estimation • Some Implementation Issues: • warping is not easy (make sure that errors in interpolation and warping are not bigger than the estimate refinement) • warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration. • often useful to low-pass filter the images before motion estimation (for better derivative estimation, and somewhat better linear approximations to image intensity) Motion estimation

  45. Optical Flow: Iterative Estimation • Some Implementation Issues: • warping is not easy (make sure that errors in interpolation and warping are not bigger than the estimate refinement) • warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration. • often useful to low-pass filter the images before motion estimation (for better derivative estimation, and somewhat better linear approximations to image intensity) Motion estimation

  46. actual shift estimated shift Optical Flow: Aliasing Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? nearest match is correct (no aliasing) nearest match is incorrect (aliasing) To overcome aliasing: coarse-to-fine estimation. Motion estimation

  47. Jw refine warp + u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image J image J image I image I Pyramid of image J Pyramid of image I Coarse-to-Fine Estimation Motion estimation

  48. Coarse-to-Fine Estimation I J refine J Jw I warp + I J Jw pyramid construction pyramid construction refine warp + J I Jw refine warp + Motion estimation

  49. Global Motion Models • 2D Models: • Affine • Quadratic • Planar projective transform (Homography) • 3D Models: • Instantaneous camera motion models • Homography+epipole • Plane+Parallax Motion estimation

  50. Least Square Minimization (over all pixels): Example: Affine Motion • Substituting into the B.C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns Motion estimation

More Related