The space of human body shapes: reconstruction and parameterization from range scans

1. The space of human body shapes:reconstruction and parameterization from range scans Brett Allen Brian Curless Zoran Popović University of Washington

2. Motivation Traditional high level characterization and sparse anthropometric measurements do not capture the detailed shape variations needed for realism PS: Sheldon, W.H. ,Stevens, S.S.,Tucker, W.B. 1940 The varieties of Human Physique ,New York

3. Motivation Traditional anthropometry has focused on sets of one-dimensional measurements.

4. ? Motivation • Full body shape capture promises to advance the • state of the art.

5. CAESAR • Civilian American & European Surface Anthropometry Resource • thousands of subjects in the U.S. and Europe • traditional anthropometry • demographic survey • laser range scans We’ll use 250 of these scans (125 male, 125 female).

6. Related work • Praun, E.,Sweldens, W,,Schroder, P. 2001 Consistent Mesh Parameterization (In proceeding of ACM SIGGRAPH 2001) • Lee, A.W.F.,Moreton, H.,Hoppe, H. 2000 Displaced Subdivision Surfaces (In proceeding of ACM SIGGRAPH 2000) • Blanz, V.,Vetter, T. 1999 A Morphable Model For The Synthesis Of 3D Faces (In proceeding of ACM SIGGRAPH 1999)

7. surface color 74 markers Scan detail • ~250,000 triangles • incomplete coverage

8. Scan detail Caused by occlusions and grazing angle views After template-based parameterization and hole filling

9. The Correspondence Problem

10. Consistent Mesh Parameterization Emil Praun Wim Sweldens Peter Schroder Princeton Bell Lab

11. Purpose and Problem Specification • Algorithm Input • Set of meshes S • Feature points F defined on each mesh M • Algorithm Goal • Determine common base domain B and connectivity L0 • Remesh each M with base domain B • Create fair patch boundaries equivalent to L0

12. Algorithms

13. Topologically Equivalent Nets • Definition: A patch is a region of semi-regular connectivity in which triangles correspond to a single triangle • Definition: A net is the outline of patch boundaries • We want a net that matches the connectivity L0 • Two patch boundaries may only intersect at a feature vertex • Each feature vertex has a consistent cyclical ordering of edges • Patch boundaries may not intersect • Naïve algorithm (shortest path) does not achieve this

14. Restricted Brush Fire Algorithm • Standard Brush Fire • Starts fire at vertex • Fire expands uniformly until it hits other vertex • That is the shortest path • Restricted Brush Fire • Uses existing paths as firewalls • Will only connect to vertex if approached from correct ordering • Must avoid blocking off vertices • Avoid completing any cycle until spanning tree of L0 traced

15. Fair Boundary Curves • Rather than simply accept topologically equivalent net, we would like certain properties • Equal distribution of surface area between patches • Smooth Patch Boundaries • Fair patch boundaries (should not swirl) • First two can be handled using relaxation • Iterative technique that involves progressively improving curve • Third requires optimization of complicated expression • Intractable, so try heuristics

16. Swirl Detection If the shortest path from {c} to {a,b} falls on the wrong side (left) ,the triangle {a,b,c} is considered flipped and may lead to swirls . On the right the trace reaches {a,b}on the correct side and the path is accepted.

17. Implementation

18. Example

19. Example

20. Matching algorithm • Find the shape that: • 1. Matches the template markers to the scanned markers 2. Moves template vertices to scanned vertices 3. Minimizes the deformation template scan

21. Match Algorithm • Applied a set of affine transformation Ti to the vertices of the template surface T and result in a new surface T’ • Minimizing three error term(data error, smoothness error, marker error)

22. Matching Algorithm

23. Objective Function • Objective Function has three weighted terms • Data error • Smoothness error • Marker error • Will use different weights in each phase of process • Multistep / Multi-resolution fitting process

24. Objective Function – Data Error • Data Error term requires current match to be close to target • Uses distance from each transformed vertex to the target surface • Weighted by confidence measure (from scanning) • Hole regions have weight = 0 • Sums total error • Distance function • Uses transformed template vertex • Takes minimum distance to “compatible” vertices in target

25. Objective Function – Smoothness Error • Measures smoothness of deformation applied to template • Problem is under-constrained using data error • E_s measures change in T_I between adjacent vertices • Encourages similarly-shaped features to be mapped to each other • Uses Frobenius norm (vector L2 norm)

26. Objective Function – Smoothness Error 補充; Frobenius norm || A ||f =

27. Smoothness Error By smoothness, we are not referring to smoothness of the deformed surface itself, but rather smoothness of the actual deformation applied to the template surface. In particular, we require afine transformations applied within a region of the surface to be as similar as possible.

28. Objective Function – Marker Error • Data and Smoothness Error can hit local minima • Example: left arm transformed to right arm • Solution: Use pre-labeled markers on the test subjects • Viewed as white dots in the range image • Correspondences set up beforehand (as in Consistent Mesh Parameterization) • 74 markers per subject (not all are used, however) • Measure distance from template marker to target marker • K_I are the indices of the markers in template, m_I are target markers

29. Algorithm Procedure • Minimize error function using L-BFGS-B algorithm • Quasi-Newton method with limited memory usage • Make four passes over data (2 low res, 2 high res) • Fit markers (low res,  = 0,  = 1,  = 10) • Refit using data term (low res,  = 1,  = 1,  = 10) • Repeat in high resolution (hi res,  = 1,  = 1,  = 10) • Refit using predominantly data term (hi res,  = 10,  = 1,  = 1)

30. Hole Filling

31. PCA (Principal component analysis) The vectors with low variance can be discarded ,and then the full data set does not to be retained in order to closely approximate the original example

32. Feature analysis Provide a direct way to explore the range of bodies with intuitive controls,such as height,weight,age and sex

33. Feature analysis

34. Markerless Matching • Using PCA, we can remove the marker requirement • A set of training data is fit using markers • Other data can be registered using unmarked range scans • Uses PCA weights to search PCA space, not transformation space

35. Markless-only We can use PCA to remove the range images • Only use markers; can be captured with much cheaper equipment • Allows us to determine approximate shape of object

36. Application 1.Transfer of Texture

37. Application 1.Morphing between model

38. Application 1.Instrumentation transfer

39. x0 x0 x0 y0 y0 y0 z0 z0 z0 x1 x1 x1 y1 y1 y1 z1 z1 z1 x2 x2 x2 average male Statistical analysis mean + PCA component #1

40. x0 x0 x0 y0 y0 y0 z0 z0 z0 x1 x1 x1 y1 y1 y1 z1 z1 z1 x2 x2 x2 average male Statistical analysis mean + PCA component #2

41. x0 x0 x0 y0 y0 y0 z0 z0 z0 x1 x1 x1 y1 y1 y1 z1 z1 z1 x2 x2 x2 average male Statistical analysis mean + PCA component #3

42. PCA reconstruction

43. Fitting to attributes

44. user constraint optimized reconstruction Fitting to points • Using the distribution of the PCA weights as a prior, we can find the most likely person that fits a set of point constraints. PCA variance

45. Summary • Contributions: • - an algorithm for creating a consistent mesh representation from range scan data. • - several ways to explore the variation in human body shape, and to synthesize and edit body models

46. Future work • - analyze shape variation between poses

47. Future work • - combine with anatomical models and physical simulation + Aubel 2003