1 / 49

Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors

Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors. Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3. 1 Carnegie Mellon University 2 Texas A&M University 3 Microsoft Research Asia. Face Poser.

octavia
Download Presentation

Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Face Poser:Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau1,3 Jinxiang Chai2 Ying-Qing Xu3 Heung-Yeung Shum3 1Carnegie Mellon University 2Texas A&M University 3Microsoft Research Asia

  2. Face Poser Generate new facial expressions with a simple and intuitive interface Inputs

  3. Face Poser Generate new facial expressions with a simple and intuitive interface Inputs Output

  4. Why Face Poser? Complex facial expressions Pre-defined controls Difficult to build and use

  5. Applications Films, Games Virtual Reality Educational

  6. Related Work Sketched-based interfaces Zeleznik et al. 96 Igarashi et al. 99 Nealen et al. 05 Kho and Garland 05 Chang and Jenkins 06 Igarashi et al. 99 Nealen et al. 05

  7. Related Work Example-based modeling Blanz and Vetter 99 Pighin et al. 99 Chai et al. 03 Zhang et al. 04 Grochow et al. 04 Sumner et al. 05 Grochow et al. 04 Sumner et al. 05

  8. Overview Database Preprocessing Model Prior

  9. Overview Database Preprocessing Model Prior User Constraints Neutral Pose Interface

  10. Overview Database Preprocessing Model Prior RuntimeOptimization User Constraints Neutral Pose New Pose Textured Pose Interface

  11. Motion capture data Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing All meshes translated and rotated to a standard view:

  12. Data: PCA representation v1x v1y v1z v2x . . . x = p is low-dimensional representation of x

  13. Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

  14. Point Constraints More detailed control User inputs: blue – 3D source vertex green – 2D target pixel Can select in any camera view

  15. Point Constraints We optimize for best p For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel

  16. Point Constraints Optimization term: Jacobian term:

  17. Point Constraints Inputs Solution

  18. Point Constraints – Results

  19. Point Constraints – Dragging interface

  20. Stroke Constraints Large-scale changes with minimal input User inputs: blue – 2D source stroke (selects 3D points on mesh) green – 2D target stroke Any curve, line, or freeform region

  21. Stroke Constraints 2D source stroke  raytrace each pixel to mesh surface to get dark gray points These can be 3D points on mesh surface (not just original mesh vertices)

  22. Stroke Constraints We optimize for best p For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke

  23. Stroke Constraints Optimization term: Jacobian term:

  24. Stroke Constraints Inputs Solution

  25. Stroke Constraints – Results

  26. Stroke Constraints – Tablet interface

  27. Stroke Constraints – Additional term If strokes are far away from each other, energy term will reach local minimum Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke Without additional term

  28. Stroke Constraints – Additional term Optimization term: Jacobian term: Without additional term

  29. Stroke Constraints – Additional term Without additional term With additional term

  30. Stroke Constraints – Results

  31. Problem statement Find best p satisfying user-constraints c: Best p is: Given a face model, how well does it match user-constraints Likelihood of face model using knowledge of data

  32. Model Priors There can be many solutions satisfying user constraints. Some of them are not realistic. We add another optimization term to constrain the solution to the space defined by the motion capture data. Without model priors term

  33. Model Priors Learn a Mixtures of Factor Analyzers (MFA) model Probability density function to measure naturalness of facial expression MFA has been applied to high-dimensional nonlinear data Without model priors term

  34. Model Priors Optimization term: Jacobian term: Without model priors term

  35. Model Priors – Result

  36. Model Priors – Result increasing weight on Model Prior term

  37. Model Priors – Result

  38. Computation time Standard PC hardware (Pentium IV 2 GHz) Point constraintstakes 0.18 seconds for 10 points time increases linearly with number of points Stroke constraintstakes 0.4 seconds for source stroke of ~900 pixels (about size of eyebrow) time increases linearly with number of pixels faster if using intermediate spline representation

  39. Cross validation New face expression samples for testing Use new samples to get target constraints Generate solution and compare with test sample

  40. Cross validation Ground truth Interpolation Optimization

  41. Comparison with other techniques Opt-blend: FaceIK [Zhang et al. 04] PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03] LWR: Locally weighted regression 3D errors

  42. Comparison with other techniques Ground truth, Optimization with PCA, Optimization with MFA

  43. Application: Trajectory Keyframing Green points – given 2D target pixels Blue points and mesh – solution

  44. Application: Trajectory Keyframing Ground truth Result

  45. Application: Trajectory Keyframing Ground truth Result

  46. Application: Trajectory Keyframing Ground truth Result

  47. Summary: Face Poser Users can learn to use our system within minutes and can create new facial expressions within seconds Inputs Output

  48. Limitation Global control • changing mouth also changes eyes • this is natural, but difficult to control sometimes Local control • changing mouth without changing eyes • but this might lead to “fake smiles”

  49. Extensions / Future work We have added different types of constraints within the same optimization framework More general: model face as separate regions, generate each region separately, and blend them back together

More Related