SIGGRAPH Course 30: Performance-Driven Facial Animation

1. SIGGRAPH Course 30:Performance-Driven Facial Animation Section: Marker-less Face Capture and Automatic Model Construction Part 1: Chris Bregler, NYU Part 2: Li Zhang, Columbia University

2. Face Tracking Approaches • Marker-based hardware motion capture systems • Tom Tolles (House of Moves) presentation 9:00 (earlier) • Parag Havaldar (Sony Pictures Imageworks) presentation at 2:15 pm

3. Marker-based Face Capture:

4. Marker-less Face Capture:

5. Early Computer Face Capture • Single Camera Input • 2D Output • Off-line • Interactive-Refinement • Make-up • Contour / Local Features • Hand Crafted • Linear Models / Tracking Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

6. Disney:

7. Early “Markerless Facecapture” • Disney: Step-Mother  Eleanor Audley

8. Early Computer Face Capture • Single Camera Input • 2D Output • Off-line • Interactive-Refinement • Make-up • Contour / Local Features • Hand Crafted • Linear Models / Tracking Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

9. Markerless Face Capture - Overview - • Single / Multi Camera Input • 2D / 3D Output • Off-line / Real-time • Interactive-Refinement / Face Dependent / Independent • Make-up / Natural • Flow / Contour / Texture / Local / Global Features • Hand Crafted / Data Driven • Linear / Nonlinear Models / Tracking

10. Common Framework Tracking = Error Minimization Error = Feature Error + Model Error

11. Difference: Tracking = Error Minimization Error = Feature Error + Model Error

12. Difference: Tracking = Error Minimization Error = Feature Error + Model Error

13. Difference: Tracking = Error Minimization Error = Feature Error + Model Error

14. Tracking = Error Minimization Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

15. Tracking = Error Minimization Error = Feature Error + Model Error

16. Tracking = Error Minimization Most general feature: Error = Optical Flow + Model Error

17. Tracking = Error Minimization Err(u,v) =  || I(x,y) – J(x+u, y+v) ||

18. Basics in Optical Flow: Lucas-Kanade 1D Image F G u ? Intensity - x Linearization: Spatial Gradient Temporal Gradient

19. Lucas-Kanade: 2D Image F G ROI ROI (u,v) SpatialGradient Temporal Gradient

20. Lucas-Kanade: Error Minimization: 2D Image Minimize E(u,v): => C D -1 C D

21. Marker-less Face Capture: In general: ambiguous using local features

22. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Optical Flow - = E(V)

23. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Optical Flow - V = E(V)

24. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Optical Flow + Model Model - V V = E(V)

25. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Optical Flow + Model Model - V V V = M( q) = E(V)

26. Optical Flow + linearized Model Model - V V 2 V = Mq Z + H V 2 Z + Cq

27. Optical Flow + 3D Model DeCarlo, Metaxas, 1999 Eisert et al 2003

28. Optical Flow + MPEG4 Model --> MediaPlayer (Eisert et al)

29. High-End Production: Optical Flow + 3D Model Disney Gemeni-Project Williams et al 2002 EA Universal Capture Borshukov et al 2002-2006

30. More “forgiving” Error Norm • - Faces change appearance L2 D

31. More “forgiving” Error Norm • L2 Norm vs Robust Norm L2 robust D D

32. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Robust Error with EM layers -

33. 2 I (1) - I(1) v t 1 I (2) - I(2) v t 2 ... I (n) - I(n) v t n Robust Error with EM layers - 0.1 0.9 0.2

34. Lucas-Kanade + changing Appearance F G - Learned PCA:

35. Optical Flow and PCA Eigen Tracking (Black and Jepson)

36. 2D texture and contours + PCA Active Appearance Models (AAM): (Cootes et al)

37. 2D texture and mesh + PCA

38. Lucas-Kanade + Apearance Models Lucas-Kanade AAMs: (Baker & Matthews)

39. Affine Flow + PCA + Robust Norm Disney: Gemeni-Project

40. Solution based on Factorization • We want 3 things: • 3D non-rigid shape model • for each frame: • 3D Pose • non-rigid configuration (deformation) • > Tomasi-Kanade-92: Rank 3 W = P S

41. Solution based on Factorization • We want 3 things: • 3D non-rigid shape model • for each frame: • 3D Pose • non-rigid configuration (deformation) • > PCA-based representations: Rank K W = Pnon-rigid S

42. Space-Time Factorization Complete 2D Tracks or Flow Matrix-Rank <= 3*K Nonrigid flow or Markerset -> “Rigid Stabilization + Blendshapes”

43. Space-Time Factorization Irani, 1999 Bregler, Hertzmann, Biermann, 2000 Torresani, Yang, Alexander, Bregler, 2001 Brand, 2001 Xiao, Kanade, 2004 Torresani, Hertzmann, 2004

44. From Pixels to 3D Blend Shapes (Torresani et al 01,02)

45. t=F . t=2 = 3D positions of point i for the K modes of deformation . t=1 . . . . . . frames wi : full trajectory Q’ mi Space-Time Tracking (Torresani Bregler 2002) Trajectory Constraints

46. From Pixels to 3D Blend Shapes (Torresani et al 01,02) • Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al) • Rank Based Tracking • 3D Basis Shapes • Probabilistic Tracking / Models • Occlusion • Dynamical Systems

47. From Pixels to 3D Blend Shapes (Torresani et al 01,02) • Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al) • Rank Based Tracking • 3D Basis Shapes • Probabilistic Tracking / Models • Occlusion • Dynamical Systems p ( I(pj,t ) | “point pj,t is visible”) = N ( I(pj,t )| µj ; 2 ) p ( I(pj,t ) | “pixel pj,t is an outlier”) = c zt = A * zt-1 + nt

48. From Pixels to 3D Blend Shapes (Torresani et al 01,02)

49. From Pixels to 3D Blend Shapes (Torresani et al 01,02)

50. Disney Gemeni Project