Robust 3D Shape Correspondence in the Spectral Domain

1. Robust 3D Shape Correspondence in the Spectral Domain Varun Jain and Hao (Richard) Zhang Graphics, Usability, and Visualization (GrUVi) Lab School of Computing Science Simon Fraser University Burnaby, BC Canada June 15, 2006

2. The correspondence problem • Given two shapes represented by triangle meshes, find a meaningful correspondence between their vertices • Not a (continuous) parameterization problem, e.g., [Kraevoy & Sheffer 04] ― min. distortion, mapped features • Applications: mesh parameterization, morphing, shape registration, tracking, recognition, and retrieval, etc.

3. Background • A classical problem studied in computer vision mostly • We are interested in fully automatic and purely shape-basedapproaches, i.e., without using prior knowledge • Goals: • Invariance to common rigid and non-rigid transformation • Robustness against noise, different object size, etc. • Ultimately, return meaningful correspondences • Despite intense studies, all proposed methods can fail on seemingly easy cases for humans

4. rotation Non-rigid ICP [Chui et al. 2004] Two basic types of techniques • Extrinsic methods • Point coordinates defined in some global frame • Optimization-based and mostly iterative, e.g., iterative closest point (ICP) • Initial alignment is crucial • Intrinsic methods • Point coordinates based on relative information • A descriptor defined from the perspective of that point • Descriptors can be absolute coordinates, e.g., spectral, or statistical, e.g., shape contexts [Belongie et al. 02]

5. Related works • With the aid of initial manual feature correspondence • Cross parameterization [Praun et al. 01, Kraevoy & Sheffer 04] • Feature-guided ICP [Sumner & Popovic 04] • Barycentric interpolation between features [Zayer et al. 05] • Other deformation based approaches • Automatic extrinsic methods: ICP and its variants • Original ICP [Besl & McKay 92] • Many variants of rigid ICP [Rusinkiewicz & Levoy 01] • Robust ICP based on refinement [Zinber et al. 03] • Non-rigid ICP based on thin-plate splines [Chui et al. 03]

6. Related works • Local shape descriptors • Shape context [Belongie et al. 02, Körtgen et al. 03] • Spin images [Johnson & Hebert 99] • Other approaches that handle rigid transformations [Gelfand et al. 05, Li & Guskov 05] • Curvature map [Gatzke et al. SMI 05] • Spectral methods • Original work on correspondence [Shapiro & Brady 92] • MDS for retrieval of isometric shapes [Elad & Kimmel 03] • Others: compression [Karni & Gotsman 00], spherical parameterization [Gotsman et al. 03], mesh sequencing [Isenberg & Lindstrom 05, Liu et al. 06], segmentation [Liu & Zhang 04], surface reconstruction [Kolluri et al. 04]

7. Spectral correspondence • [Shapiro & Brady, 92]: Given two point sets P and Q • Construct symmetric “affinity” matrices AP and AQ using pair-wise L2 distances and a Gaussian kernel • Construct spectral embedding by kleading eigenvectors of AP and AQ, sorted by descending eigenvalues • Compute best matching using embedded coordinates via L2 distance in the k-dimensional spectral domain • Why spectral correspondence? • Affinities are intrinsic measure (but high-dimensional) • Eigenvectors have good approximation properties • Spectral embeddings normalize shapes with respect to all rigid body transformations and uniform scaling

8. Key observations • Flexibility of affinity measures • Whichever transformation one needs the correspondence to be invariant of, build that invariance into affinities • Eigenvectors need to be scaled properly, e.g., at least to handle data with difference sizes • Eigenvectors can “switch” (never reported before) • Signs of eigenvectors need to be consistent • Non-rigid shape transformation can cause non-rigid transformation in the spectral domain

9. Summary of our approach • Use geodesic affinities for invariance to shape bending • Eigenvector scaling using squared root of eigenvalues • Proper handling of objects with difference sizes • Eigenvalue decay leaves approach less sensitive to k • Variance-normalization + interpretation from kernel PCA • Heuristics to resolve eigenvector switch and sign flip • Non-rigid ICP via thin-plate splines in spectral domain • Net result: • Proper correspondence of articulated shapes • Consistently more robust correspondence results

10. Evaluation paradigm • Visual examination via color plots • Manually color one model based on parts • Color second model using computed correspondence • Plot of percentage of correct matches • Manually provided ground truth (small feature sets) • Ground truth automatically identified via “in-place” mesh decimation • Plot of correspondence error • Sum of correspondence errors at the vertices • Error at a vertex: geodesic distance between ground truth and computed corresponding point

11. Eigenvector scaling Geodesic-based spectral embedding Best matching Non-rigid ICP via thin-plate splines Basic steps of our algorithm

12. Geodesic affinities • Given two meshes M1 and M2 of sizes n1 and n2 • Affinity matrices A1 (n1n1) and A2 (n2n2) given by where d1 and d2 are geodesic distances on M1 and M2 • Gaussian: importance of far-away vertices attenuated • Gaussian width set as maximum geodesic distance • Other kernel, e.g., exponential, may be applied

13. k-dimensional spectral embedding coordinates of ith the point of mesh Spectral embeddings • Eigen-decompose each affinity matrix A = UUT • Obtain k-D spectral embedding of mesh vertices using the k leading (scaled) eigenvectors of A • First eigenvector ignored as it is almost a constant

14. Examples: 3D spectral embeddings Use of 2nd, 3rd, and 4th scaled eigenvectors

15. Eigenvector scaling • EkEkTgives the best rank-k approximation of the affinity matrix A (namely, dot product  affinity) • Scaling using the square root of the eigenvalues is shown to normalize the variance of data • The scaling is also a natural one when seen from the perspective of kernel PCA [Jain 2006]

16. Eigenvector switching and sign flips • Signs of the eigenvectors are decided by eigensolver and are difficult to correspond automatically • Discrepancy between shapes can also cause certain eigenvectors to switch places • An eigenvector switch or a sign flip corresponds to a reflection in the spectral domain

17. Exhaustive search and greedy heuristic • Reflection-invariant shape descriptors possible, e.g., high-D shape context or symmetric polynomials, but more invariance  less descriptivity [Frome et al. 04] • Choose among 2kk! possible eigenvector ordering and sign flips to minimize a correspondence cost: • Besides exhaustive search (for very small k), can use greedy scheme to order one eigenvector at a time

18. Non-rigid transformations • Perform non-rigid ICP using thin-plate splines in the spectral domain • Experimentally, very fast convergence (5-10 iterations)

19. Eigenvector scaling Geodesic-based spectral embedding Best matching Non-rigid ICP via thin-plate splines Recap of algorithm

20. Result: % of correct correspondences Manual initial alignment used for first three! % is out of 17-20 ground-truth matches (200-300 vertices; k = 5 eigenvectors used throughout)

21. Visual results for correspondence Models with articulation and moderate stretching Many more results in color plate (page 300).

22. Limitations • Intrinsic geodesic affinities • Symmetry issue • Topological issue: hybrid approach [Jain & Zhang 06] • Rather primitive heuristic for resolving eigenvector switching and sign flips • Effectiveness attributed to spectral normalization • Euclidean metric as correspondence cost • No particular reason, except for a computational one • Challenge: what is right? • Computational complexity: O(n2logn)

23. Follow-up and future works • Sampling via Nyström approximation [Liu et al. 06] • Spectral embedding: O(n2logn)  O(pnlogn + p3) • Little loss of quality at low sampling (10 out of 4000) • Farthest point sampling used • More sophisticated sampling schemes [Liu & Zhang 06] • Retrieval of articulated shapes [Jain & Zhang 06] • Outperforms light-field descriptor [Chen et al. 03] and spherical Harmonics descriptor [Kazhdan et al. 03] (even when these are applied to spectral embeddings) • But not so on Princeton Benchmark database (yet) due to various artifacts in the models • How about eigenspaces?

24. Acknowledgement • NSERC Grant 611370 • MITACS Grant on project: Mathematical Surface Representations for Conceptual Design • MATLAB code of non-rigid ICP [Chui et al. 03] • Greg Mori for helpful discussions • Reviewers’ comments and for pointing out a couple of missing references Thank you!