Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors

1. Face Poser:Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau1,3 Jinxiang Chai2 Ying-Qing Xu3 Heung-Yeung Shum3 1Carnegie Mellon University 2Texas A&M University 3Microsoft Research Asia

2. Face Poser Generate new facial expressions with a simple and intuitive interface Inputs

3. Face Poser Generate new facial expressions with a simple and intuitive interface Inputs Output

4. Why Face Poser? Complex facial expressions Pre-defined controls Difficult to build and use

5. Applications Films, Games Virtual Reality Educational

6. Related Work Sketched-based interfaces Zeleznik et al. 96 Igarashi et al. 99 Nealen et al. 05 Kho and Garland 05 Chang and Jenkins 06 Igarashi et al. 99 Nealen et al. 05

7. Related Work Example-based modeling Blanz and Vetter 99 Pighin et al. 99 Chai et al. 03 Zhang et al. 04 Grochow et al. 04 Sumner et al. 05 Grochow et al. 04 Sumner et al. 05

8. Overview Database Preprocessing Model Prior

9. Overview Database Preprocessing Model Prior User Constraints Neutral Pose Interface

10. Overview Database Preprocessing Model Prior RuntimeOptimization User Constraints Neutral Pose New Pose Textured Pose Interface

11. Motion capture data Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing All meshes translated and rotated to a standard view:

12. Data: PCA representation v1x v1y v1z v2x . . . x = p is low-dimensional representation of x

13. Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

14. Point Constraints More detailed control User inputs: blue – 3D source vertex green – 2D target pixel Can select in any camera view

15. Point Constraints We optimize for best p For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel

16. Point Constraints Optimization term: Jacobian term:

17. Point Constraints Inputs Solution

18. Point Constraints – Results

19. Point Constraints – Dragging interface

20. Stroke Constraints Large-scale changes with minimal input User inputs: blue – 2D source stroke (selects 3D points on mesh) green – 2D target stroke Any curve, line, or freeform region

21. Stroke Constraints 2D source stroke  raytrace each pixel to mesh surface to get dark gray points These can be 3D points on mesh surface (not just original mesh vertices)

22. Stroke Constraints We optimize for best p For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke

23. Stroke Constraints Optimization term: Jacobian term:

24. Stroke Constraints Inputs Solution

25. Stroke Constraints – Results

26. Stroke Constraints – Tablet interface

27. Stroke Constraints – Additional term If strokes are far away from each other, energy term will reach local minimum Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke Without additional term

28. Stroke Constraints – Additional term Optimization term: Jacobian term: Without additional term

29. Stroke Constraints – Additional term Without additional term With additional term

30. Stroke Constraints – Results

31. Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

32. Model Priors There can be many solutions satisfying user constraints. Some of them are not realistic. We add another optimization term to constrain the solution to the space defined by the motion capture data. Without model priors term

33. Model Priors Learn a Mixtures of Factor Analyzers (MFA) model Probability density function to measure naturalness of facial expression MFA has been applied to high-dimensional nonlinear data Without model priors term

34. Model Priors Optimization term: Jacobian term: Without model priors term

35. Model Priors – Result

36. Model Priors – Result increasing weight on Model Prior term

37. Model Priors – Result

38. Computation time Standard PC hardware (Pentium IV 2 GHz) Point constraintstakes 0.18 seconds for 10 points time increases linearly with number of points Stroke constraintstakes 0.4 seconds for source stroke of ~900 pixels (about size of eyebrow) time increases linearly with number of pixels faster if using intermediate spline representation

39. Cross validation New face expression samples for testing Use new samples to get target constraints Generate solution and compare with test sample

40. Cross validation Ground truth Interpolation Optimization

41. Comparison with other techniques Opt-blend: FaceIK [Zhang et al. 04] PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03] LWR: Locally weighted regression 3D errors

42. Comparison with other techniques Ground truth, Optimization with PCA, Optimization with MFA

43. Application: Trajectory Keyframing Green points – given 2D target pixels Blue points and mesh – solution

44. Application: Trajectory Keyframing Ground truth Result

45. Application: Trajectory Keyframing Ground truth Result

46. Application: Trajectory Keyframing Ground truth Result

47. Summary: Face Poser Users can learn to use our system within minutes and can create new facial expressions within seconds Inputs Output

48. Limitation Global control • changing mouth also changes eyes • this is natural, but difficult to control sometimes Local control • changing mouth without changing eyes • but this might lead to “fake smiles”

49. Extensions / Future work We have added different types of constraints within the same optimization framework More general: model face as separate regions, generate each region separately, and blend them back together