Use and Re-use of Facial Motion Capture

1. Use and Re-use of Facial Motion Capture M. Sanchez, J. Edge, S. King and S. Maddock

2. 0. Motivation • Facial Animation in the Computer Graphics industry is a mainly human-driven process, requiring a lot of time and resources • Other aspects of Character Animation, such as Skeletal Animation, have been successfully automated by the use of human motion capture technology • Applying the same approach to Facial Animation could sensibly reduce the workload involved, and would lead to a corresponding increase in the realism of Facial Animation • However, the differences in the nature of the captured content requires the development of specific techniques to make Facial Motion Capture applicable

3. 1. System overview

4. 1.1 Facial Motion Capture • System input: 3D tracking of a predefined set of markers attached to the skin surface Problems: • It doesn’t analyze the motion over the full geometry of the face (just at the markers); • The captured face may not correspond with the face to be animated; • Noise and missing data.

5. 2. Animating the skin

6. 2. Animating the skin • How to reconstruct the deformation of the complete skin surface when only the movement of a few points is known? • Direct interpolation of the movement of the markers over the skin (Kshirsagar et al. 00, Pasquariello and Pelachaud. 01) • Dirichlet Free Form Deformations (Escher et al. 98) • Radial Basis Functions (Fidaleo, Noh et al. 00) • We use Planar Bones (Sanchez and Maddock 03)

7. 2.1. Planar Bones • Extended formulation of Surface-oriented Free Form Deformations (Kokkevis and Singh, 00) • Define a parameterisation of every vertex over a control mesh, used to drive a deformation • Preserve a “distance relation” between the control structure and the deformed geometry • Replicate proper transmission of motion across the skin without the need of surface metrics

8. 4. Retargeting Facial Motion Capture • The dimensions of the face are different, and so is the scale of the motion; • Conventional full-body Motion Capture retargeting is not applicable; • The correspondence between different faces is highly non-linear.

9. 4. Building a mapping between faces

10. 4. Building a mapping between faces • Retargeting FMC requires: • Adapting the Planar Bones control mesh to the target geometry; • Scaling the motion of the markers according to the change of physiognomy. • Ideally, both processes should be performed automatically • In practical terms, we need some user input.

11. 4.1. Fitting the control mesh

12. 4.1. Fitting the control mesh • 3 stages: • Radial Basis Functions - produce initial approximation • Cylindrical projection – Constraining the markers to the target surface • Mesh fitting – Blind constrained optimisation • Additional parameters of the Planar Bones method are also retargeted by this process: • Extents of the deformation (affection volume) • Discontinuity maps

13. 4.1. Fitting the control mesh • RBF stage: • Build an interpolant of the offset between the markers labelled on the target face and their equivalents in the reference model • Evaluate this function at the non-hand-labelled markers to obtain their image on the target geometry • Mesh fitting stage: • Finding the optimal distribution of control points that: • Minimizes the “distance” between the reference face deformed by the retargeted control mesh and the target geometry • Preserves the general shape represented by a deformation energy function • Stays on the surface of the target face (enforced through the cylindrical mapping) Simplex downhill method

14. 4.2. Scaling Facial Motion

15. 4.2. Scaling Facial Motion • The two faces are labelled with the same markers After fitting the control mesh • We can extend this mapping to the whole space the faces are given in By interpolating the initial displacement at every control point using RBFs • This interpolant is used to compute the mapping on the target space of the captured markers during the animation

16. 4.2. Scaling Facial Motion • This procedure implicitly scales the movement of the markers in the target space, according to the initial correspondence that is given as reference. An evaluation the 2-norm of the metric tensor of the mapping shows how infinitesimal displacements are scaled • green: positive scaling (>1) • blue: negative scaling • The Planar Bones algorithm computes the final deformation, driven by the retargeted control mesh

17. 5. Processing Motion Capture Input • Limitations in the marker tracking technology lead to deficiencies in the captured data:

18. 6. Sample results: lip tracking

19. 6. Sample results: lip tracking

20. 7. Conclusions and future work • We have introduced a novel method for the retargeting and animation of faces from motion capture data • Current research: • Provide a better model for the tracking of the inner contour of the lips: • Marker-less image processing of the video capture • Physical model using a mass-spring system attached to the outer contour • Introduce furrowing and wrinkling in the skin animation • A posteriori deformation analysis on the deformation induced by Planar Bones

21. Questions? m.sanchez @dcs.shef.ac.uk j.edge @dcs.shef.ac.uk

22. Additional samples: chorus