Recovering High Dimensional Non-Rigid Deformations Through Point Matching

1. Recovering High Dimensional Non-Rigid Deformations Through Point Matching Haili Chui Image Processing and Analysis Group Departments of Electrical Engineering Yale University

2. Alignment of Deformable Objects Associated by Non-Rigid Deformations

3. Point Matching Tool: Point Matching

4. Outline • Non-rigid point matching: • Problem. • Algorithm. • Search and match strategy. • Global-to-local search strategy. • Examples. • Applications: • Key-frame animation. • Face matching/warping. • Average shape estimation. • Brain MRI image registration.

5. * High dimensional parameter space. * Hard optimization problem. Non-rigid Point MatchingProblem Difficulties: * Noise. * Outliers. Correspondence Transformation

6. Related Work • Rigid point matching methods. • Can not be extended into non-rigid matching. • Non-rigid point matching methods. • Iterative Closest Point matching (ICP), graph matching, modal matching, shape context matching, active shape model and etc. • Various shortcomings: • Need extra information (e.g. curve, needs point ordering information). • Not general (e.g., only allow one specific kind of deformation). • Not accurate (e.g., methods designed for object recognition only result in rough alignment / correspondence). • Not robust (e.g., the methods can be easily upset by bad initialization, existence of noise, presence of spurious points (outliers)).

7. Our Design Goal • A robust algorithm capable of handling the general non-rigid point matching problem. • Least assumptions: only requires point coordinates. • Complete: correspondence + non-rigid transformation. • General: allow room for different kinds of transformations. • Accurate: achieve matching from coarse level to fine details. • Robust: • Tolerate noise. • Able to reject outliers. • Not sensitive to bad initializations. • Fast and simple to implement.

8. Developing a Robust Point Matching Algorithm • Examination of the variables: • Correspondence + Non-Rigid Transformation. • Joint optimization framework. • Search and match strategy. • Global-to-local search strategy. • Better models for outliers and regularization. • Final energy function and algorithm.

9. Outlier row and column Variable I: Correspondence • Given two point sets: • Match matrix • Inner • All rows and columns (except outlier) sum up to 1,  1-to-1 correspondence.

10. Match Matrix (M) • Correspondence: 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 Outlier 0 0 0 1 Outlier

11. Variable II: Transformation • Spatial mapping function: • Specified by transformation parameters: • Incorporating prior constraints (e.g. smoothing constraint) by adding a penalty measure to the energy function.

12. Joint Optimization Formulation • Non-rigid point matching problem: • Linear assignment + least squares problem. • Under constraints:

13. Search and Match Strategy • Interesting relationship between the Correspondence and the Transformation. • Search and match strategy: • Fix transformation, estimate correspondence. • Fix correspondence, estimate transformation. • Essentially a grouped descent algorithm. • Still going to be plagued by local minima.

14. How to Overcome the Local Minima? • Fuzzy correspondence. • A more flexible correspondence model. • Deterministic annealing (DA). • Control the “fuzziness” of the correspondence. • Accomplish matching: • global-to-local. • coarse-to-fine.

15. Fuzzy Correspondence • Conventional correspondence: • Only allow binary values, 0 or 1. • Complete commitment to the fittest / closest (e.g., ICP) matching candidate. • Fuzzy correspondence: • Allow continuous values, between [0,1]. • Partial, multiple matches. • Intuition: when the matching is far from optimal, it doesn’t make sense to commit for one local candidate. • Conventional correspondence is the limiting case of fuzzy correspondence.

16. Search Range T controls Fuzziness of Correspondence Less Fuzzy Search locally Fuzzy Search globally T T T Fuzzy CorrespondenceVisualization

17. By controlling T, accomplish global-to-local (coarse-to-fine) matching. • Gaussian form, • T high, M  fuzzy (uniform), allow matching to far away points. • T low, M  binary, allow matching to only nearby points,. Deterministic Annealing (DA) • How: add an entropy measure term, • Encourage the correspondence to be fuzzy. • The “temperature” parameter T weighs such fuzziness.

18. Gradually reduce T Deterministic Annealing Global-to-Local Matching Strategy Update Correspondence Low T End High T Initialize Slowly reduce T Update Transformation DA: A Global-to-Local Matching Strategy Observation: T Controls Fuzziness of Correspondence T Controls Search Range High T Extremely Fuzzy Correspondence Search Globally Low T Less Fuzzy Correspondence Search Locally

19. Incorporate a specific non-rigid transformation model, Thin Plate spline (TPS), to form a complete algorithm. • TPS-RPM Algorithm. Incorporating A Specific Transformation Model • Designed as a general framework, RPM can accommodate different transformation models. • General mapping function: • General regularization:

20. Examples of RPM

21. Example I • A simple example to help visualizing the global-to-local match strategy.

22. Deformed Template Significant Correspondence Deformed Space Annealing Circles Example I: Matching Process

23. Example I: Matching Process

24. Example II: • Handle both noise and outliers.

25. Example II: Matching Process

26. A Comparison: ICP and RPM • Similarity: • Jointiterativealternating update method. • ICP: • Binary correspondence. • Nearest neighbor heuristic. • Easily trapped by local minima. • Outlier rejection: dynamic thresholding. • Poor model for outliers. • RPM: • Fuzzy correspondence. • Deterministic annealing: global-to-local matching strategy. • Less likely to be trapped by local minima. • Directly models outliers. • Better model to handle outliers.

29. Applications

30. Point Matching Example Application: Key-Frame Animation Interpolation

31. Example Application: Key-Frame Animation

32. Point Matching Example Application: Face Matching

33. Example Application: Face Matching

34. Estimating An Average Shape • Given multiple sample shape (sample point sets), compute the average shape for which the joint distance between the samples and the average is the shortest. Average ? • Difficult if the correspondences between the sample points are unknown.

36. Application: Brain Anatomical Feature Registration (MRI Image)

37. Distribution Before Alignment Direct Comparison of Subjects Brain Function Image Alignment of Subjects Distribution After Alignment Comparison of Subjects After Alignment Studying Function-Structure Connection

38. Inter-Subject Brain Registration • Inter-subject brain registration: • Alignment of brain MRI images from different subjects to remove some of the shape variability. • Difficulties: • Complexity of the brain structure. • Variability between brains. • Brain feature registration: • Choose a few salient structural features as a concise representation of the brain for matching. • Overcome complexity: only model important structural features. • Overcome variability: only model consistent features.

39. Feature Extraction Feature Fusion Outer Cortex Surface Point Feature Representation Feature Matching All Features Subject I Major Sulcal Ribbons Point Feature Representation Subject II A Unified Feature Registration Method

40. Application: Brain Shape Normalization

41. Two types of feature investigated: • Outer cortex surface. • Major sulcal ribbons. • Comparison of different methods: Method I Method II Method III Comparison of Different Features • Different features can be used in our approach.

42. Results: Method I vs. Method III • Outer cortical surface alone can not provide adequate information for sub-cortical structures. • Combination of two features works better.

43. Results: Method II vs. Method III • Major sulcal ribbons alone are too sparse --- the brain structures that are relatively far away from the ribbons got poorly aligned. • Combination of two features works better.

44. Conclusion • Unified brain feature registration approach: • Combination of different features improves registration. • Benefits: general/unified, symmetric, efficient. • Robust point matching algorithm: • Capable of estimating non-rigid transformations and correspondences. • Robust (noise/outliers/initializations). • Non-rigid point matching can be a general tool for many different problems.

45. 3D Surface Intensity Image Matching Intensity Images Feature Point Matching 3D Point Nodes with Attributes Future Work • Extend point matching to the matching of intensity images. • Intensity-based registration and feature-based registration have been viewed as totally different and largely unrelated approaches. • In fact, they can be related to each other. • Extend to segmentation: non-rigid point shape model + intensity information.

46. Future Work • Better model for non-rigid deformation. • Incorporate more information (e.g., attributes) into the point representation. • Interesting connection between point matching and graph matching.

47. Acknowledgements • Thesis advisor and committee members: • Anand Rangarajan, James Duncan, Hemant Tagare and Peter Schultheiss. • IPAG members: • Lawrence Staib, Xiaolan Zeng, Xenios Papademetris, Oskar Skrinjar, Ravi Bansal, Yongmei Wang, Pengcheng Shi, James Rambo, Gang Liu, Ning Lin, Xiaoning Qian, Zhong Tao and Jing Yang. • Colleagues in the brain registration project: • Robert Schultz, Lawrence Win and Joseph Walline. • Financial support is provided by the grants from the Whitaker Foundation, NSF, NIH. • Further information: http://noodle.med.yale.edu/~chui/.

48. RPM Detail

49. Joint Optimization Formulation • Non-rigid point matching problem: • Linear assignment + least squares problem. • Under constraints:

50. By controlling T, accomplish global-to-local (coarse-to-fine) matching. • Gaussian form, • T high, M  fuzzy (uniform), allow matching to far away points. • T low, M  binary, allow matching to only nearby points,. Deterministic Annealing (DA) • How: add an entropy measure term, • Encourage the correspondence to be fuzzy. • The “temperature” parameter T weighs such fuzziness.