SIGGRAPH Course 30: Performance-Driven Facial Animation

1. SIGGRAPH Course 30:Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: http://cs.nyu.edu/~bregler/sig-course-06-face/

2. SIGGRAPH Course 30:Performance-Driven Facial Animation Section: Markerless Face Capture and Automatic Model Construction Part 1: Chris Bregler, NYU

3. Markerless Face Capture

4. Markerless Face Capture - Overview - • Single / Multi Camera Input • 2D / 3D Output • Real-time / Off-line • Interactive-Refinement / Face Dependent / Independent • Make-up / Natural • Flow / Contour / Texture / Local / Global Features • Hand Crafted / Data Driven • Linear / Nonlinear Models / Tracking

5. Markerless Face Capture – History – • Single Camera Input • 2D Output • Off-line • Interactive-Refinement • Make-up • Contour / Local Features • Hand Crafted • Linear Models / Tracking Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

6. Tracking = Error Minimization Err(u,v) =  || I(x,y) – J(x+u, y+v) ||

7. Tracking = Error Minimization In general: ambiguous using local features

8. Tracking = Error Minimization Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

9. Tracking = Error Minimization Error = Feature Error + Model Error

10. Tracking = Error Minimization Error = Optical Flow + Model Error

11. Optical Flow (Lucas-Kanade) I J v ? i Intensity - x 2 I(x ) - J(x + v ) E(V) i i i 2 I (x ) - I(x ) v linearize D i i i t

12. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Optical Flow + Model Model - V V D = E(V) D D

13. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Optical Flow + Model Model - V V D V = M( q) = E(V) D D

14. Optical Flow + linearized Model Model - V V 2 V = Mq Z + H V 2 Z + Cq

15. Optical Flow + Hand-Crafted Model DeCarlo, Metaxas, 1999 Williams et a,l 2002

16. Optical Flow and PCA Eigen Tracking (Black and Jepson)

17. PCA over 2D texture and contours Active Appearance Models (AAM): (Cootes et al)

18. PCA over 2D texture and contours

19. PCA over texture and 3D shape 3D Morphable Models (Blanz+Vetter 99)

20. Affine Flow and PCA

21. 3D Model Acquisition - Multi-view input: Pighin et al 98

22. Solution for Rigid 3D Acquisition Structure from Motion: - Tomasi-Kanade-92 3D Pose 3D rigid Object Factorization

23. Acquisition without prior model ? • No Model available ? • Model too generic/specific ? • Stock-Footage only in 2D ?

24. Solution based on Factorization • We want 3 things: • 3D non-rigid shape model • for each frame: • 3D Pose • non-rigid configuration (deformation) • > Tomasi-Kanade-92: Rank 3 W = P S

25. Solution based on Factorization • We want 3 things: • 3D non-rigid shape model • for each frame: • 3D Pose • non-rigid configuration (deformation) • > PCA-based representations: Rank K W = Pnon-rigid S

26. 3D Shape Model Linear Interpolation between 3D Key-Shapes: S S S 2 1

27. Basis Shape Factorization Complete 2D Tracks or Flow Matrix-Rank <= 3*K

28. Nonrigid 3D Kinematics from point tracks

29. Nonrigid 3D Kinematics from dense flow

30. Nonrigid 3D Kinematics from dense flow

31. Nonrigid 3D Kinematics from dense flow

32. Nonrigid 3D Kinematics from dense flow Synthesis Modeling Motion Capture

33. Markerless Face Capture - Summary - • Single / Multi Camera Input • 2D / 3D Output • Real-time / Off-line • Interactive-Refinement / Face Dependent / Independent • Make-up / Natural • Flow / Contour / Texture / Local / Global Features • Hand Crafted / Data Driven • Linear / Nonlinear Models / Tracking