Real-Time Enveloping with Rotational Regression

1. Real-Time Enveloping with Rotational Regression Robert WangKari PulliJovan Popović

2. Enveloped (skinned) characters are pervasive. skeleton mesh Skeletons are often used to control meshes.

3. Physically based modeling provides realistic deformations. • Realistic deformations • Finite-element based [Teran et al. 2005] • Anatomy based [Scheepers et al. 1997] • Elastically deformable [Capell et al. 2002, 2005] • Used in movie production • Off-the-shelf commercial tools • Slow evaluation [Teran et al. 2005] [Absolute Character Tools 1.6]

4. We learn a fast model from exported examples. Black Box Simulation Exported Examples (skeleton-mesh pairs) Fast Model Our method

5. Artists can still use existing modeling tools or scanned data. 3-D Scan Data Exported Examples (skeleton-mesh pairs) Fast Model

6. mesh simplification learning Physical simulation High-resolution mesh Low-resolution mesh Rotational Regression Enveloping This is analogous to mesh simplification. • Higher quality • Used in movie production • Faster to render • Optimized for interactive applications

7. ? How do we map a skeleton to a mesh? What parameters should we learn? How to model muscle deformations for fast evaluation?

8. + y y Linear blend skinning linearly maps joint rotations to vertex positions. • Most popular enveloping technique for games • Coarse modeling parameters (but very simple) • Not very expressive (but very fast) Figure from [Wang and Phillips 2002]

9. Linear blend skinning has many names. • Also known as, • Single-Weight Enveloping • Skeletal Subspace Deformation (SSD) • Or just, “Skinning” • We will use “Linear Blend Skinning” or “SSD.”

10. The two steps of our work are deformation gradients prediction and mesh reconstruction. Deformation gradients prediction (Rotational Regression) Mesh reconstruction

11. + We present a replacement for linear blend skinning. • Coarse modeling parameters. • Can’t handle certain types of deformations. • Fast • Lets you use your existing modeling tool. • Good for muscle bulges. • Fast Whenever you have an existing model, you should use our technique instead of linear blend skinning.

12. Our model is inspired by the behavior of a flexing bicep. • Rigid components move with the bone rotation • Other surfaces rotate in the opposite direction Surface rotation Bone rotation

13. Angle is scaled by u. Axis is offset by rotation W. target rotation (surface) source rotation (bone)

14. We map a sequence of bone rotations to a sequence of surface rotations. … … target rotation sequence (surface) source rotation sequence (bone)

15. We fit parameters uand Wby regression. Surface rotations u’,W’ Best-fit parameters Skeleton rotations

16. Rotational regression is good at capturing muscle bulges.

17. Deformation gradients prediction Mesh reconstruction stitches deformation gradients together. Mesh reconstruction

18. C D(q) Mesh reconstruction solved with least-squares. • Least-squares problem equivalent to linear system. • Computation is matrix-multiplication. deformation gradients vertex positions Least –squares

19. Near-rigid vertices help eliminate low-frequency errors at extremities. • Low-frequency errors can accumulate at extremities of mesh • We fix a set of near-rigid vertices to their SSD predictions • Still a least squares problem

20. We build upon existing mesh reconstruction work. • Mesh IK [Sumner et al. 2005], [Der et al. 2006] • SCAPE [Anguelov et al. 2005] • Similar formulation, faster evaluation. [Anguelov et al. 2005]

21. C Dk(q) Here’s a review of what we’ve covered. Rotational Regression Deformation Gradients Prediction Least-squares problem Mesh Reconstruction

22. Model reduction lowers the dimensionality of problem. • Large multiplication on CPU C  C’  Dl(q) Dk(q) SSD • Smaller multiplications on GPU

23. P = 450 225 110 50 25 Model reduction uses greedy clustering. • Vertices clustered into proxy-bones. • Per-triangle deformation gradients clustered into “key” deformation gradients.

24. Mesh reconstruction reduced to the following matrix-multiplications. All on GPU: • Computation in fragment program C’  Dl(q) Map from “key” deformation gradients to proxy-bones “key” deformation gradients SSD weights

25. Skinning Mesh Animations is a an alternative approach to model reduction. • The method from Skinning Mesh Animations uses mean-shift clustering and is more robust to errors. [James and Twigg 2005] • Our method minimizes vertex error and is faster

26. Deformation gradients prediction is now on “key” deformation sequences. • Fewer deformation gradient sequences to predict rotational regression.

27. Mesh reconstruction step reduced to matrix-multiplications on GPU. • Smaller matrix-multiplications • Supported on graphics hardware C’  Dl(q)

28. Our Technical Contributions: Rotational Regression Accurate and GPU-Ready Poisson Reconstruction Model Reduction

29. Results

30. Results

31. Results

32. Our work approximates the training examples better than SSD and also generalizes well.

33. Our model is suitable for interactive techniques. • Evaluation speed within a factor of two of SSD • Off-line training preprocess is usually less than half an hour

34. How does our work fit with previous work?

35. Our work is complementary to displacement correcting techniques. Figure from [Kry et al. 2001] • Previous work provide corrective displacements. • Pose space deformation[Lewis et al. 2000], • Shape by example [Sloan et al. 2001], • Eigenskin [Kry et al. 2002] • Our work provides better approximation of rotations. • Our work complements approaches that build upon SSD.

36. Displacement correcting approaches fail when SSD is very wrong.

37. Our work builds upon previous ideas on enriching the SSD model. • Multi-weight enveloping [Wang and Phillips 2002] • Additional joints [Mohr and Gleicher 2003] • Our technique has more parameters than SSD and generalizes the additional-bones model.

38. A more expressive model is useful here.

39. Our model doesn’t do a perfect job. • Not perfect reproduction • Inspired by muscle bulging and twisting. Other behaviors empirically validated. • Displacement correcting technique can be used for exact reproduction of examples.

40. Conclusion: Fast and accurate enveloping. • Fast evaluation of physical simulations through learning. • Within a factor of two of SSD on most models • Accurate reproduction of details • Better approximation and generalization • Complementary to previous work • A replacement for linear blend skinning

41. Funding Nokia Research Center National Science Foundation Pixar Animation Studios Hardware/Software NVIDIA Corporation Autodesk Data Drago Anguelov Joel Anderson Michael Comet, Comet Digital, LLC Mark Snoswell, CG Character Joseph Teran, Ron Fedkiw MIT Graphics Group Ilya Baran Jiawen Chen Sylvain Paris Acknowledgements

42. Questions? • Thank you for coming to our talk!

43. Learning tasks trade expressiveness and simplicity. Simpler: Easier to fit Fewer training examples needed. Less likely to overfit. More Expressive: Captures more types of deformation. Rotational Regression

44. Linear blend skinning (SSD) is a rough and ready map from joint rotation matrices to vertex positions. • Most popular enveloping technique for games • Coarse modeling parameters (but very simple) • Not expressive enough (but very fast) desired deformation SSD deformation

45. Model Reduction • True optimization not as tractable • We approximate it with a greedy algorithm inspired by mesh simplification. difficult to solve simultaneously discrete optimization

46. Our work builds upon previous ideas on • Additional joints [Mohr and Gleicher 2003] • Multi-weight enveloping [Wang and Phillips 2002] • Our technique generalize the additional- bones model • We evaluate cross-validation error to test for overfitting [Wang and Phillips 2002]

47. Rotational regression is good at capturing muscle bulges.