Recovering Articulated Object Models from 3D Range Data

1. Recovering Articulated Object Models from 3D Range Data Dragomir Anguelov Daphne Koller Hoi-Cheung Pang Praveen Srinivasan Sebastian Thrun Computer Science Dept. Stanford University

2. Problem overview Data: Articulated object observed in different configurations 3D range surfaces (meshes) obtained by a scanner Task: Recover the articulated object model Register all surfaces Cluster the object surface into rigid parts Recover joints linking the parts Applications of articulated models Animation, object detection, scene segmentation

3. Surface registration 2 1 3 • Algorithm input: meshes Z0, Z1, … ZN • Pick template mesh X = Z0 • Need to recover point-to-point correspondences between X and instance meshesZi , i={1, …, N} • Any registration algorithm providing can be used • Correlated Correspondences algorithm[Anguelov et al ‘04] • Unsupervised nonrigid registration algorithm • Handles large deformations, occlusion (but not clutter) • Algorithm Overview • Computes embedding of mesh Z into mesh X defined as a joint assignment to correspondences • Encodes correspondence correlations (minimizing deformation, preserving geodesic distances) as pairwise potentials in a Markov network over • Uses Loopy-BP to obtain a good joint assignment to all variables

4. Clustering into rigid parts:Generative model a1 aM … Transformations x1 xM T … y1 yM rotation translation … b1 bk z1 zk … Point labels Points 1. Rigid part points share the same rigid transformaition in each instance Part labels Template X Points 2. Each part is a connected region of the template mesh surface Transformed Model a) Soft contiguity constraints: If and are connected in X Instance Z j = Part (xj) k = ModelPoint (zk) for b) Hard contiguity constraints: Each part consists of a single connected component of the surface of X Assume  values provided by a nonrigid registration algorithm

5. Clustering into rigid parts: Optimization • Model Summary: • Ignoring hard contiguity constraints, generative model defines a Markov Network over the part labels : • Objective • Iterative Optimization • E-step: Given transformations , compute the part labels maximizing Eq. (1) and satisfying the hard contiguity constraints • M-step: Given part labels , compute transformations Point in mesh Zi , generated by template point xj

6. Clustering into rigid parts: E-Step • The problem of finding a MAP assignment to given which maximizes Eq. (1) can be expressed as an integer program (IP). A relaxation of the IP produces a linear program(LP)[Kleinberg et al ’99]. • For 2-part models, the optimal LP solution is also the optimal IP solution. • For n-part models, n>2: • In case of a fractional LP solution, a rounding procedure produces an IP solution within a factor of 2 of the optimal. • The LP can be solved very efficiently by using multiway min-cuts [Kolmogorov et al ’04] • The LP solution may not satisfy hard contiguity constraints. • A part consisting of m connected components is split into m separate parts which doesn’t change the objective in Eq. (1).

7. Clustering into rigid parts: LP formulation • Introduce variables s.t. • Log-cost associated with potential becomes where • Define separation cost of an edge as where • Putting everything together: Relax binary constraints to obtain LP

8. Clustering into rigid parts: M-step and Annealing • M-step: find rigid part transformations given part labels • Optimization function decomposes into a separate equation for each : • Eq. (2) is isomorphic to the ICP problem [Besl’ 92]. Close form estimate of is obtained by solving a simple LP. • Annealing • Number of parts in solution depends on the tradeoff where large values of result in fewer parts • When estimate of is poor and is large, solving E-step LP is hard, because multiple plausible solutions with similar scores exist • Starting with low values of and gradually increasing performs best in practice

9. Clustering into Rigid Parts: Initialization • Need good initial values for the transformations • Method 1: Cluster local rigid transformations [Cheung et al ’03] • Associate a vector of transformations with each point (estimate using the known correspondences for all instance meshes) • Cluster points into k parts using adaptive PCA [Archer et al ’02] • Slow:>10 mins for 2000 point models Template X b) a) • Method 2: Break surface into k arbitrary fragments of approximately equal size • Fast:<1s for 2000 point models • Same final results! a) Rigid parts after initialization b) Rigid parts after EM

10. Optimizing the Joint Positions • A joint is a point belonging to two parts simultaneously: [Cheung et al ’03] • To solve for the joint, must optimize • Sometimes, the solution can be an entire space of points • Example: a joint with one degree of freedom • Introduce an additional regularization term, enforcing that the joint is close to the boundary between two parts in X Joint between parts p and q b) a) a) Rigid parts after EM b) Estimated joints

11. Results: Puppet Dataset Correctly identifies all 15 puppet parts and the skeleton

12. Results: Initialization • Testing EM Initialization • Initialize with a different number of parts k • If k > 15, the algorithm consistently finds the correct number of parts (15)

13. Results: Arm Dataset Correctly identifies 4 arm parts and the skeleton in the presence of non-rigid deformation

14. Results: Annealing • Results of annealing with increasing value of • Increasing decreases the number of parts (A – D) • Increasing too much results in a partition which tries to reduce the number of links between parts (E)

15. Conclusions and Future Work • Algorithm for recovering articulate object models from 3D data • Clusters the object surface into rigid parts • Recovers the articulated object skeleton • Algorithm properties • Probabilistic formulation • Efficient global inference • Algorithm recovers complex models consisting of a large number of parts in real world datasets • Future work • Extend the algorithm to allow and explicitly model (small) deformations of the articulated parts Robustness to local maxima