240 likes | 380 Views
Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis. Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3 1 University of Washington 2 Industrial Light & Magic 3 University of Toronto. movies. games. telepresence.
E N D
Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis Brett Allen1,2, Brian Curless1, Zoran Popović1, and Aaron Hertzmann3 1 University of Washington 2 Industrial Light & Magic 3 University of Toronto
movies games telepresence design Motivation
+ Identity pose shape Goal • We would like to be able to generate body models of any individual in any pose. - want to synthesize models in real-time - model should be learnable from real data
Data • CAESAR data set: 44 subjects in 2 poses • Multi-pose data set: 5 subjects in 16 poses • Dense-pose data set: 1 subject in 69 poses [Anguelov et al. 2005]
Anatomical methods Chadwick et al. 1989 Turner and Thalmann 1993 Scheepers et al. 1997 Wilhelms and Van Gelder 1997 … Aubel 2002
v = shape vector p = example parameters = function parameters Example-based methods Given: n examples v0…vn and n sets of parameters p0…pn (optional) find .
1 2 2 1 3 4 4 3 Allen et al. 2002 Scattered data interpolation columns of = key shapes i = reconstruction weights from applying k-nearest neighbors or RBFs on p Lewis et al. 2000 Sloan et al. 2001 Kry et al. 2002
An aside on enveloping Enveloping + scattered data interpolation = “corrective enveloping”
Blanz & Vetter 1999 Latent variable models x = latent variable (component weights) W = components in columns v = average shape Allen et al 2003 Seo et al 2003 Anguelov et al 2005 …
Pose variation vs body variation Sloan et al. 2001
Pose variation vs body variation Anguelov et al. 2005
Intrinsic skeleton parameters: bone lengths and carrying angles Our approach c = “character vector”: all information needed to put a character in any pose v = shape in a particular pose
Two Problems 1. Scans might not be at “key” poses 2. Scans are not complete Maximize: p(c | {e()}) …actually, we don’t know the pose or skinning weights either: Maximize: p(c, s, q{} | {e()})
Going to multiple characters • One possibility: Learn several character vectors separately, then run PCA. • Two problems: • the character vector contains values that have different scales (rest positions, offsets, bone lengths) • we don’t have enough data!
Identity variation ~ ~ c is the character vector of the th example person g(c,s,q) applies skinning and pose space deformations
Alternating optimization • We initialize the {x} with the weights from running PCA on estimated skeleton parameters. • We then optimize W, c, s, q. • Then we optimize for {x}. • Repeat…
Results (video)
Conclusions We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two. + very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves + the learned model can generate body shapes very quickly (over 75 fps)
Limitations • You need a lot of data! Our data set was too sparse in some areas. • Some poses are hard to capture. • It’s very hard to compensate for the skinning artifacts. • The shape matching could be improved (high-frequency details are lost if the matching is poor).
Acknowledgements • UW Animation Research Labs • Washington Research Foundation • National Science Foundation, NSERC, CFI • Microsoft Research, Electronic Arts, Sony, Pixar • Kathleen Robinette and the AFRL lab • Dragomir Anguelov • Domi Pitturo