Kernel-based tracking and video patch replacement

1. Kernel-based trackingand video patch replacement Igor Guskov guskov@eecs.umich.edu

2. Overview • Research areas • Geometry processing • Compression of geometry • Feature-based matching • Template matching in video

3. Projects • Geometry processing • Semi-regular remeshing • Parameterization • Add structure to meshes • Do wavelet compression • Dynamic mesh compression • Soft-body animations • Extract • Do wavelet compression

4. Projects • Matching • 3D matching • Automatic scan alignment • Shape recognition • Tracking non-rigid geometry in video • For geometry reconstruction • Real-time reconstruction • For video editing and surveillance

5. Find approximate alignment automatically Registration: ICP Optimal alignment Approximate alignment Approximate Surface Alignment Joint work with Xinju Li

6. Video tracking • Feature tracking • Classical approach: Lucas&Kanade tracker • Based on mean-square error minimization • We want to track larger patches

7. Tracking features • Point features • Given point-to-point correspondences • Can do reconstruction of 3D geometry, many other things • Linear features • Track stick figures: limbs • Reconstruct articulated characters • Recognize activity • Silhouettes • Patch features • Active appearance models (AAMs) • Geometry + texture + appearance • Face tracking • Video editing: monet from Imagineer Systems

8. Error-based tracking • Mean-square error • Image I(x) • Template T(y) • Warp map x=W[z](y) • For instance: • W[p](y) = y+p • Small patch translated around • W[(,t)](y) = y+t • Translation + uniform scaling • W[h](y) = h(y) • Homography h minp || I(W[p](y)) – T(y) ||2 T(y) I(x)

9. Quad-marked surface tracking • Collection of quads • SCA 2003 • Real-time trackingand reconstruction • Four cameras

10. Mean-shift tracking • Formulate tracking as mean-shift problem • Comaniciu, Ramesh, Meer CVPR 2000 • Replace a pixel by the distribution of color values in a neighborhood • Histogram • Best match of a histogram • Robust to noisy data • Very fast algorithm

11. Histogram matching • Bhattacharya coefficient (p,q) • Given two distributions p(z) and q(z) • Related to bounds on the probability of classification error between these two distributions • P(error) ≤ (p,q) • For matching, we want P(error)=1

12. Distance between distributions • Metric space of histograms • Not that important in the original paper • Implement as a simple sum

13. Where is mean-shift? • The way the histograms are computed • Weighted histograms • Pixels at the blob center contribute more • Setting the gradient of Bhattacharya coefficient to zero one gets • Each pixel contributes its opinion on how relevant it is to be the center of the blob

14. Mean-shift clustering • Comaniciu, Meer PAMI 2002 • Kernel density estimation • Sum of bumps of width h

15. Extensions • Previous work • Translation + scale [Collins 03] • Particle-tracking [Perez et al 02] • Multiple collaborating trackers [Hager et al 04] • Template alignment • More general warps • Warp is the key • Translation does not really warp • Need to account for that properly

16. Templates I • Multiple blobs tracked together • Each has its own histogram pk[t] • Easy to do by considering squared sum of distances

17. Templates II: warp • Where is that weighted histogram coming from? • Random variable X • Displacement from the blob’s center • Histogram bin pa • With translation • General warp

18. Triangles • Affine warps • Six parameters • Cannot account for perspective distortion • Okay for weak perspective • Multiple triangles needed • Relations among the collection of triangles • Multiscale

19. A formula • Histogram bin value Jacobian of the inverse warp All the pixels yin the imagewhich fall into bin a Warp the pixel positionback into canonical spaceand take its probability density

20. Simple illumination model • Cannot rely on colors being constant • Illumination changes • Outdoors: clouds etc. • Shadowing • Cameras set on automatic exposure • Always collect relative colors • Average illumination locally L(x) • Histogram of I(W(X))-L(W(X)) • This requires some texture to be present Roll-ball video

21. Optimization • Bhattacharya coefficient • Take the gradient w.r.t. z • Explicit formula • Feed to the optimization library

22. Implementation • YUV video • Histogram in two channels out of three • Y is luminance • Higher resolution • UV is color • Histograms 16x16 bins • Templates have 120 blobs (16*15/2)

23. Results • Videos • About one second per frame • Extend to masked template

24. Video augmentation • Previous work • Bartoli & Zisserman 2004 • RBF estimation & grid • Pilet et al. 2005 • Keypoint features • Real-time detection • Lin 2005 • Near-regular textures

25. User input • Masks for tracking and replacement • Tracking of the templates • Warping of the replacement grid • Poisson edit on the replacement region

26. Warping the grid • Blend affine transformations • Warping of the replacement grid

27. Masks and grids

28. Replacement image • Select replace • Poisson edit

29. Motion blur • Necessary for visual quality • Smear the replacement region • Perform Poisson gradient fitting in a larger region

30. Results • Videos

31. Conclusions • Basic tracking procedure • Imperfect match • Non-rigid patches • Large areas • Replacement in videos • Simple user input • Warping and Poisson edit