Learning Models of Shape from 3D Range Data

1. Learning Models of Shape from 3D Range Data Dragomir Anguelov Artificial Intelligence Lab Stanford University

2. Shape Models for Animation

3. Animation Biomechanics Fz Mz Mx Fy Fx My [Gollum - Time Warner] Shape Models for Motion Estimation

4. Shape Models for Scene Understanding Goal: Understand sensor input in terms of objects and relations “puppet holding stick”

5. Simulation-based models [Aubel ‘02] [Wilhelms, Van Gelder ‘97] Example-based models [Allen et al. ‘02] [Allen et al. ‘03] Machine Learning for Model Construction Artist-designed models [Poser – Curious Labs] [Lucasfilm] [Dreamworks]

6. Marker motion capture Markerless motion capture Physical measurement [Bregler et al. ‘98] [Braune,Fischer 1892] [Polar Express] [Cheung et al ’03] Machine Learning for Motion Estimation

7. Shape Models from 3D Scans • Object models: • Discover object parts • Model pose variation in terms of parts • Class models: • Model shape variation within class Pose variation Body-shape variation

8. 3D Range Scans • Cyberware Scans • 4 views, • ~125k polygons • ~65k points each • Problems • Missing surface • Drastic shape changes

9. Articulated Template 2. Fit Template to Scans 3. Interpolation Standard Modeling Pipeline A lot of human intervention Pose or body shape deformations modeled, but not both [Allen, Curless, Popovic 2002] Similar to: [Lewis et al. ‘00] [Sloan et al. ’01] [Mohr, Gleicher ’03], …

10. 3D Scans Registration Recover skeleton Learn model of deformations Contributions Unsupervised non-rigid registration Automatic articulated model recovery Modeling pose and body shape deformations 3D Scan segmentation and object detection(not in thesis)

11. Talk outline • Automating the data processing pipeline • Unsupervised non-rigid registration • Recovering an articulated model • Modeling the space of human deformations • Scene understanding (not in thesis)

12. Registration Task: Establish correspondences between two surfaces

13. C Correspondences Transformation Q Model mesh X Transformed mesh X’ Correspondence ck specifies which point x’i generated point zk Generative Model Scan meshZ Goal: Given model mesh X and scan mesh Z, recover transformation Q and correspondences C

14. Q Transformation Gaussian noise model Deformation Model Treat each link as a spring, resisting stretching and twisting Model X

15. Non-rigid Iterative Closest Point • Algorithm • Assume initial alignment Qknown • Compute correspondences C (given Q) • For each point zk , find its nearest neighbor x’i • Solve for transformation Q (given C) which • Brings matching pairs together • Minimizes deformation [Shelton ’00], [Chui & Rangarajan ’02], [Allen et al. ’03], [Hähnel et al.’03]

16. Correspondences for different points computed independently c1 c2 Poor correspondences Poor transformations Nonrigid ICP Experiment X Z

17. Z X Correlated Correspondence Algorithm Z X Correlated Correspondence Algorithm Computes an embedding of mesh Z into mesh X Output: Correspondences Input: Pair of scans • The embedding enforces: • • Minimal surface deformation • Similar local surface appearance • Preservation of geodesic distance [Anguelov, Srinivasan, Pang, Koller, Thrun, Davis ‘04]

18. Single potential Pairwise potential (C1, C2) C1 C2 (C2) (C1) (C3) (C1,C3) (C2,C3) C3 • Joint probability distribution Markov Network • Markov network

19. Model Point x2 Model Point x1 Link Link Local appearance Local appearance Deformation potential Appearance potential 1 2… N (Ci,Cj) Ci Cj (Ci) (Cj) Scan Point zj Scan Point zi Local appearance Local appearance Correlated Correspondence Model

20. (C1, C2) C1 C2 (C2) (C1) (C3) (C1,C3) (C2,C3) C3 CC Potentials • Markov network Want a good consistent assignment for all correspondences C ! • Local appearance potentials (Ci=k) • Use spin images [Johnson+Hebert ’97] to quantify the surface similarity around two matching points • Deformation potentials D(Ci=k,Cj=l) = P(e’ij| ekl)

21. (C1, C2) C1 C2 (C2) (C1) (C3) (C1,C3) (C2,C3) C3 Markov Network Inference • Markov network • Inference is Markov Nets is generally intractable • Exponential search space • Loopy Belief Propagation (LBP) [Pearl ’88] is an efficient algorithm for search in exponential spaces • Converges to a local minimum (of the Bethe free energy)

22. Nearby points in Z must be nearby in X Constraint between each pair of adjacent points zi, zj Scan Z Model X Z X Geodesic Potentials: near -> near

23. Geodesic Potentials: far -> far Z X • Distant points in Z must be distant in X • Constraint between each pair of distant points zk, zl (farther than 5r) r resolution of mesh X

24. Results: Pose Deformation No markers used

25. Results: Body Shape Deformation No markers used

26. Application: Scan Completion Cyberware scans Model Registrations • 4 markers were placed manually on each of these scans

27. Applications: Animation Linear interpolation in local link deformation space

28. Talk outline • Automating the data processing pipeline • Unsupervised non-rigid registration • Recovering an articulated model • Modeling the space of human deformations • Scene understanding (not in thesis)

29. Recovering articulated models Input: models, correspondences Output: rigid parts, skeleton [Anguelov, Koller, Pang, Srinivasan, Thrun ‘04]

30. combine Skeleton: 9 parts Recovering Articulation: State of the art • Algorithm assigns points to parts independently; ignoring the correlations between the assignments • Prone to local minima Each joint is estimated from a separate sequence [Cheung et al., ‘03]

31. Recovering articulation [Anguelov et al. ’04] • Stages of the process • Register meshes using Correlated Correspondences algorithm • Cluster surface into rigid parts • Estimate joints

32. Recovering articulation [Anguelov et al. ’04] • Stages of the process • Register meshes using Correlated Correspondences algorithm • Cluster surface into rigid parts • Estimate joints

33. 1 2… P Transformations T1 a1 aN Part labels … Model … xN x1 Points … TP Transformed Model yN y1 … cK c1 Point corrs Instance zK z1 … Points Probabilistic Model

34. a1 a2 a3 Contiguity Prior • Parts are preferably contiguous regions • Adjacent points on the surface should have similar labels • Enforce this with a Markov network: Penalizes large number of parts

35. Clustering algorithm • Algorithm • Given transformations , perform min-cut* inference to get • Given labels , solve for rigid transformations If a part doesn’t contribute to the likelihood, it will be automatically dropped *[Greig et al. 89], [Kolmogorov & Zabih 02]

36. Clustering Movie

37. Results: 70 Human Scans Tree-shaped skeleton found Rigid parts found

38. Results: Puppet

39. Results: Arm

40. Application: Tracking [Anguelov, Mündermann, Corazza ‘05]

41. Application: Tracking [Mündermann, Corazza, Anguelov ‘05]

42. Application: Tracking [Mündermann, Corazza, Anguelov ‘05]

43. Talk outline • Automating the data processing pipeline • Unsupervised non-rigid registration • Recovering an articulated skeleton • Modeling the space of human shapes • Pose and body shape deformations • Application: shape completion • Scene understanding (not in thesis) • Discriminative Markov networks for scan segmentation • Articulated object detection

44. [Allen et al. ‘03] [Seo & Thalmann‘03] Body shape deformation = point displacements from average shape Deformation Transfer Problem [Lewis et al. ‘00] [Sloan et al. ‘01] [Allen et al. ‘02] [Wang & Phillips ‘02] [Mohr & Gleicher ‘03] [Sand et al. ‘03] Pose deformation = point displacements from articulated template How do you combine two displacement-based models? • displacements cannot be multiplied • adding displacements ignores notion of object scale

45. Deformed polygon Template polygon Pose deformation Body shape deformation Rigid part rotation Predict from nearby joint angles Linear subspace (PCA) Predicting Human Deformation [Anguelov, Srinivasan, Koller, Thrun, Rodgers, Davis ‘05]

46. Deformed polygon Template polygon Pose deformation Body shape deformation Rigid part rotation We have: To reconstruct the entire mesh Y, solve: Reconstructing the Shape Related work: [Sumner & Popovic ’04]

47. Joint angles Deformations output Regression function Linear regression from two nearest joints Pose Deformation input

48. Pose Deformation Space

49. output Low-dimensional subspace (PCA) Body Shape Deformation input

50. Body Shape Deformation Space