Symmetry-based Segmentation and Recognition

1. Symmetry-based Segmentation and Recognition Thomas B. Sebastian Brown University

2. Summary • Three main contributions of my dissertation: • Effective segmentation method for carpal bones in CT images • Implements skeletal coupling between seeds • Solves convergence problem of deformable models • Generic curve matching algorithm with several applications • Robust shape recognition algorithm for matching shock graphs • Gives excellent recognition rates for indexing large shape databases

3. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its applications • Shape recognition using shock graphs • Indexing into large shape databases

4. Medical application • Study 3D kinematics of carpal bones • Identify wrist injuries where radiographs are normal • Quantify shape of bone to characterize disease progression, e.g., in Kienbock disease • Compute curvature maps

5. Challenges • Weak and diffused edges • Gaps in bone boundary

6. Challenges (cont.) • Texture in spongy bone • Narrow inter-bone gaps

7. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its application • Shape recognition using shock graphs • Indexing into large shape databases

8. Deformable models Initialize multiple seeds, which grow under image-dependent forces • Drawbacks • Fail to converge near weak edges • Do not capture narrow inter-bone gaps

9. Seeded region growing [Biscoff PAMI] • Initializes seeds and grows them by annexing an adjacent pixel • Only the "closest" pixel is added at each iteration • Implements global competition among all seeds • Drawbacks • Leaks into small gaps as there are no geometric constraints

10. Region Competition [Zhu/Yuille, PAMI] • Initializes seeds and grows them using a combination of statistical and smoothing forces • Implements local competition between seeds when they contact each other • Allows for exchange of pixels between regions and recovery from errors

11. Region Competition (cont.) • Drawbacks • Merges some adjacent bones • Fails to capture low-contrast bones

12. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its application • Shape recognition using shock graphs • Indexing into large shape databases

13. Skeletal coupled deformable model (SCDM) • SCDM combines advantages of previous techniques • Subpixel nature of deformable models • Global competition of seeded region growing • Local competition of region competition

14. Skeletal coupling: Overview

15. Growth of regions • In isolation, growth of regions depends on a local statistical force

16. l d Local competition by skeletal coupling • The inter-region skeleton is used to couple growing regions • The regions compete for pixels in the middle • When seeds come in contact, l=1, same as region competition

17. Long-range skeletal competition • The inter-region skeleton is viewed as predicted boundary and the growth of regions is modulated by its suitability

18. Total skeletally-coupled forces • The total statistical force is a combination of local and long-range forces

19. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its application • Shape recognition using shock graphs • Indexing into large shape databases

20. Convergence of SCDM • SCDM solves the convergence problem of traditional deformable models

21. SCDM segmentation results Results are clinically meaningful

22. 3D model of carpal bones 3D model is created by stacking up 2D contours 3D model is useful in studying carpal kinematics, creating computational atlases, etc.

23. Carpal bone segmentation: Comparison SRG/RegComp SCDM • Rating by hand surgeons (RIH) • SCDM 83% • SRG 14% • RegComp 3%

24. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its applications • Shape recognition using shock graphs • Indexing into large shape databases

25. Curve matching [Sebastian et al, PAMI] • Goal is to find optimal alignment (pairing of points) and distance (deformation cost) • Cost is measured by length and curvaturedifferences of infinitesimal segments and summing them

26. Curve matching [Sebastian et al, PAMI] • Alignment is a pairing of points of C, C’ • Treat curves C and C’ as the x and y-axes • Alignment is then a curve in 2D space, called alignment curve • Dynamic programming to find optimal alignment curve

27. Prototype formation • Average curve can be computed by averaging corresponding sub-segments

28. Shape morphing • Weighted averages can be used to generate morph sequence

29. Handwritten character recognition • 327 characters (34 categories) • 98.5 % recognition rate

30. Gesture recognition Intuitive matches

31. Gesture recognition (cont.) Outperforms other methods Precision Recall

32. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its applications • Shape recognition using shock graphs • Indexing into large shape databases

33. Real Example Shock graph representation of shapes • Shocks (or medial axis or skeleton) are locus of centers of maximal circles that are bitangent to shape boundary Shape boundary Shocks

34. Distance between shapes • Shape space is collection of all shapes • Shape is a point • Shape deformation sequence is a path • Cost of optimal deformation sequence is distance from A to B • There are infinitely many deformation paths • Shape space has to be discretized

35. Changes in shock graph topology • Shock graph topology is unaltered for most shape deformations • At transition shapes small changes lead to abrupt changes in the shock graph topology

36. Partitioning of shape space • Shape cell: Collection of shapes with same shock graph topology • Transition shapes form the boundary between shape cells Shape cell 2 Shape cell 1

37. Discretization of deformation paths • Shape deformation bundle:Collection of deformation paths passing through the same set of shape cells • Represented by set of transition shapes it passes through Dynamic programming is used to find optimal edit sequence

38. Matching results* [Sebastian et al, ICCV 01] Edit-distance algorithm gives intuitive results * Same colors indicate matching edges; gray-colored edges are pruned

39. Robustness to transformations Boundary noise In optimal edit sequence “noisy” branches are pruned Articulation Edit-distance is robust in presence of part-based changes

40. Robustness to transformations (cont.) Viewpoint variation • Deform edit handles smooth changes • Splice and contract edits handle abrupt changes

41. Robustness to transformations (cont.) Partial occlusion Edit-distance is robust to partial occlusion

42. Indexing into shape databases* Edit-distance algorithm allows 100% shape recognition between different shape categories * Results duplicated in two databases: 99 shapes and 216 shapes

43. Indexing Results It also allows nearly 100% recognition between shapes in the same shape category

44. Contents • Segmentation using skeletally coupled deformable model • Introduction • Previous approaches • Skeletal coupling • Results • Curve matching and its applications • Shape recognition using shock graphs • Indexing into large shape databases

45. Indexing into large databases [ECCV 02] • Indexing results are excellent using a large database (1032 shapes, 200 exemplars) • Correct category is selected in top 1, 2 and 5 matches with 78%, 91%, and 99% success rate respectively

46. Number of exemplars • Using fewer exemplars suggests a hierarchical representation • 2-3 primary exemplars rule out 75% of categories • Additional 2-3 auxiliary exemplars rule out another 50% of categories

47. Acknowledgments • Collaborators/Advisors • Prof. Benjamin Kimia, Brown University • Prof. Philip Klein, Brown University • Dr. Joseph Crisco, RI Hospital

48. Selected References • SCDM segmentation • Segmentation of carpal bones from CT images using SCDM [MedIA02] • Segmentation of carpal bones using SCDM [MICCAI98] • Curve matching • On aligning curves [PAMI02] • Alignment-based recognition of shape outlines [IWVF01] • Constructing curve atlases [MMBIA00] • Shock-graph matching • Recognition of shapes by editing their shock graphs [ICCV01] • Shape matching using edit distance: An implementation [SODA01] • Indexing into databases • Shock-based indexing into large databases [ECCV02] • Metric-based shape retrieval in large databases [ICPR02] • Curves vs. skeletons for object recognition [ICIP01]