Parametric Motion and Tracking

1. CS 636 Computer Vision Parametric Motion and Tracking Nathan Jacobs

2. the plan • Questions and concerns? • Recap • Parametric motion • Layered motion Slides for today from Robert Pless and Ming-Hsuan Yang

3. review from last time… • Motion fields • Optical flow • Lucas-Kanade Point Tracking • Iterative Lucas-Kanade

4. Examples of Motion Fields (a) (b) (a) Motion field of a pilot looking straight ahead while approaching a fixed point on a landing strip. (b) Pilot is looking to the right in level flight. Computer Vision, Robert Pless

5. Examples of Motion Fields (a) (b) (c) (d) (a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background. Computer Vision, Robert Pless

6. The brightness constancy constraint • How many equations and unknowns per pixel? • One equation, two unknowns • Intuitively, what does this constraint mean? • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown gradient (u,v) • If (u, v) satisfies the equation, so does (u+u’, v+v’) if (u’,v’) (u+u’,v+v’) edge

7. Lucas-Kanade flow • How to get more equations for a pixel? • Basic idea: impose additional constraints • most common is to assume that the flow field is smooth locally • one method: pretend the pixel’s neighbors have the same (u,v) • If we use a 5x5 window, that gives us 25 equations per pixel!

8. Iterative Refinement • Iterative Lucas-Kanade Algorithm • Estimate velocity at each pixel by solving Lucas-Kanade equations • Warp H towards I using the estimated flow field • - use image warping techniques • Repeat until convergence

9. estimate update Initial guess: Estimate: Optical flow: iterative estimation x x0 (usingd for displacement here instead of u)

10. estimate update Initial guess: Estimate: x x0 Optical flow: iterative estimation

11. estimate update Initial guess: Estimate: Initial guess: Estimate: x x0 Optical flow: iterative estimation

12. Optical flow: iterative estimation x x0

13. Affine Perspective 3D rotation Translation 2 unknowns 6 unknowns 8 unknowns 3 unknowns Motion models

14. Example: affine motion • Substituting into the B.C. equation Each pixel provides 1 linear constraint in 6 global unknowns Least square minimization (over all pixels):

15. Parametric motion i.e., the product of image gradient with Jacobian of the correspondence field

16. Parametric motion the expressions inside the brackets are the same as the cases for simpler translation motion case

17. Other 2D Motion Models

18. Generalizing Lucas-Kanade extends to all differentiable warp functions! Lucas-Kanade 20 Years On: A Unifying Framework, Simon Baker and Iain Matthews

20. to matlab… to explore linear image approximations % load data im = im2double(rgb2gray(imread('peppers.png'))); [gxgy] = gradient(im); sz = size(im); [X Y] = meshgrid((1:sz(2)) -sz(2)/2, (1:sz(1)) -sz(1)/2); motionModel = 'translation'; motionModel = 'affine'; % make jacobian images % i x j x Np x 2 switchmotionModel case 'translation’ Np = 2; imJacobian = zeros([szNp 2]); imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; case 'affine’ Np = 6; imJacobian = zeros([szNp 2]); imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; imJacobian(:,:,3,1) = X; imJacobian(:,:,4,1) = Y; imJacobian(:,:,5,2) = X; imJacobian(:,:,6,2) = Y; end imSteepest = sum(bsxfun(@times, imJacobian, cat(4, gx, gy)), 4); tmpJacobian = scale2rgb(repmat(reshape(imJacobian, [sz 1 Np*2]), [1 1 3 1])); tmpSteepest = scale2rgb(repmat(reshape(imSteepest, [sz 1 Np]), [1 1 3 1])); figure(1); clf; montage(tmpJacobian, 'Size', [2 Np]) title({'Jacobian Images: change in the warpfieldw.r.t. parameters'}) figure(2); clf; montage(tmpSteepest, 'Size', [1 Np]) title('SteepestDescent Images: innerproductwith image to update parameters') [Tx Ty] = ndgrid(linspace(-1, 1, 20), linspace(-1,1,20)); p = randn(100, Np); p = bsxfun(@times, p, [1 1 .01 .01 .01 .01]); for ix = 1:size(p,1) % make the approximation using the steepest descent images (the first % order approximation of the image) figure(3); clf; subplot(121); imagesc(im + sum(bsxfun(@times, imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off; title(p(ix,:)) subplot(122) imagesc(im + sum(bsxfun(@times, imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off;xlim([40 100]), ylim([160 210]); colormap gray pause(eps) end

21. Motion representations How can we describe this scene?

22. Block-based motion prediction • Break image up into square blocks • Estimate translation for each block • Use this to predict next frame, code difference (MPEG-2)

23. Layered motion • Break image sequence up into “layers”:  = • Describe each layer’s motion

24. Layered motion • Advantages: • can represent occlusions / dis-occlusions • each layer’s motion can be smooth • video segmentation for semantic processing • Difficulties: • how do we determine the correct number? • how do we assign pixels? • how do we model the motion?

25. Layers for video summarization

26. Background modeling (MPEG-4) • Convert masked images into a background sprite for layered video coding + + + + + + =

27. What are layers? • Layers or Sprites • Alpha matting • Intensities • Velocities • Wang and Adelson, “Representing moving images with layers”, IEEE Transactions on Image Processing, 1994

28. How do we form them?

29. How do we form them?

30. How do we estimate the layers? • compute coarse-to-fine flow • estimate affine motion in blocks (regression) • cluster with k-means • assign pixels to best fitting affine region • re-estimate affine motions in each region…

31. Layer synthesis • For each layer: • stabilize the sequence with the affine motion • compute median value at each pixel • Determine occlusion relationships

32. Results

33. Why is are layers useful?

34. summary • parametric motion: generalized Lucas-Kanade • layered motion models

35. for next time • starting 3D, structure from motion • SzeliskiChap 7