Contours and Optical Flow: Cues for Capturing Human Motion in Videos

1. Contours and Optical Flow: Cues for Capturing Human Motion in Videos Thomas Brox Computer Vision and Pattern Recognition Group University of Bonn Research partially funded by the German Research Foundation (DFG)

2. Human pose tracking from video • Tracking of markers attached to the body + Designed to be easy to track  Reliable and fast tracking • Accuracy limited by number of markers • People may feel uncomfortable • Tracking features that naturally appear in the images • Patches (e.g. KLT, SIFT, etc.) • Contour/Silhouette • Optic flow How to extract these features reliably from the images • Introduction • Segmentation • Optic Flow • Summary

3. Contour and optic flow based human tracking • Introduction • Segmentation • Optic Flow • Summary Joint work with Bodo Rosenhahn

4. Part I Object Contour Extraction

5. Object contour extraction • Find two regions: object & background • Often: Static background background subtraction • Optimality criteria here: • Strong similarity within regions • Small boundary • Bayesian approach: • Introduction • Segmentation • Optic Flow • Summary

6. Level set representation of contours(Dervieux-Thomasset 1979, Osher-Sethian 1988) • Introduce embedding function • Contour C represented as zero-level line of • Introduction • Segmentation • Optic Flow • Summary Courtesy of Daniel Cremers

7. Region-based active contours (Chan-Vese 2001, Paragios-Deriche 2002) H(x) H’(x) • Minimize negative logarithm: • Gradient descent:plus update of p1 and p2 • Introduction • Segmentation • Optic Flow • Summary

8. Region statistics • 7 channels: • 3 color channels (CIELAB) • 4 texture channels • Channels assumed to be independent • Probability densities pijapproximated by Gaussians • Introduction • Segmentation • Optic Flow • Summary

9. Texture • Usually modeled by Gabor filters(Gabor 1946) • Includes • Magnitude • Orientation • Scale • High redundancy • Sparse alternative representation feasible • Nonlinear structure tensor (Brox et al. 2006) • Region based local scale measure(Brox-Weickert 2004) • Introduction • Segmentation • Optic Flow • Summary

10. Sparse texture features  Gabor filter bank Sparse representation • Introduction • Segmentation • Optic Flow • Summary

11. Examples for contour extraction • Introduction • Segmentation • Optic Flow • Summary

12. Local region statistics • Object and background usuallynot homogeneous • Idea: assume them to be locally homogeneous • Probability densities estimated by local Gaussians • Introduction • Segmentation • Optic Flow • Summary

13. Introducing a shape prior • Idea: object model can serve as 3-D shape prior • Constrains the segmentation, unwanted solutions discarded • Bayesian formula: • Pose parameters of model unknown  Two variables: contour and pose • Introduction • Segmentation • Optic Flow • Summary

14. Joint optimization • Simultaneously optimize contour and pose: • Iterative alternating scheme: • Update contour • Update pose parameters • Related works: 2-D shape priors (Leventon et al. 2000, Cremers et al. 2002, Rousson-Paragios 2002) conventional segmentation part shape+pose constraint • Introduction • Segmentation • Optic Flow • Summary

15. Part II Optic Flow

16. Optic flow based tracking • Introduction • Segmentation • Optic Flow • Summary Image 1 and 2, estimate flow in between Given pose at Image 1 Estimated pose at Image 2 Pose change due to optic flow

17. Tracking example • Introduction • Segmentation • Optic Flow • Summary

18. How to compute the optic flow? ? • Given: two images I(x,y,t) and I(x,y,t+1) in a sequence • Goal: displacement vector field (u,v) between these images • Variational approach: (Horn-Schunck 1981) • Introduction • Segmentation • Optic Flow • Summary

19. Enhanced model(Brox et al. 2004, Papenberg et al. 2006) Robust smoothness term(Cohen 1993, Schnörr 1994) Robust data term(Black-Anandan 1996, Mémin-Pérez 1996) Gradient constancy (Brox et al. 2004) Non-linearized constancy (Nagel-Enkelmann 1986, Alvarez et al. 2000) Spatiotemporal smoothness (Nagel 1990) Original Horn-Schunck: • Introduction • Segmentation • Optic Flow • Summary Final optic flow model:

20. Impact of each improvement Horn-Schunck Robust data term Gradient constancy Nonlinear constancy Spatiotemporal smoothness Robust smoothness ü ü ü ü ü ü • Introduction • Segmentation • Optic Flow • Summary Correct result

21. Accurate and robust optic flow computation • Introduction • Segmentation • Optic Flow • Summary

22. Contour and optic flow based human tracking • Introduction • Segmentation • Optic Flow • Summary Joint work with Bodo Rosenhahn

23. Summary • Contours and optic flow can be reliable features for pose tracking • Texture, local statistics, and a shape prior are important for general contour based human motion tracking • High-end optic flow helps in case of fast motion What’s next? • Real-time performance • Automatic pose initialization • Prior knowledge about joint angle configurations • Introduction • Segmentation • Optic Flow • Summary

24. Outlook • Introduction • Segmentation • Optic Flow • Summary Joint work with Bodo Rosenhahn

25. Backup: nonlinear structure tensor • Texture orientation can be measured with the structure tensor (second moment matrix)(Förstner-Gülch 1987, Rao-Schunck 1991, Bigün et al. 1991) • Gaussian smoothing  nonlinear diffusion • Introduction • Segmentation • Optic Flow • Summary Input image Nonlinear structure tensor Linear structure tensor

26. Backup: region based local scale measure • Estimate regions, measuretheir size • Nonlinear diffusion: TV flow(Andreu et al. 2001) • Tends to yield piecewise constant images  regions • Local evolution speed inverselyproportional to size of region(Steidl et al. 2004) local scale measure Input image • Introduction • Segmentation • Optic Flow • Summary Local scale