Recognizing Deformable Shapes

1. Recognizing Deformable Shapes Salvador Ruiz Correa (CSE/EE576 Computer Vision I)

2. Goal • We are interested in developing algorithms for recognizing and classifying deformable object shapes from range data. 3-D Output Surface Mesh 3-D Laser Scanner Range data Post- processing Input 3-D Object (Cloud of 3-D points) • This is a difficult problem that is relevant in several application fields.

3. Applications • Computer Vision: - Scene analysis - Industrial Inspection - Robotics • Medical Diagnosis: • Classification and • Detection of craniofacial deformations.

4. Basic Idea • Generalize existing numeric surface representations for matching 3-D objects to the problem of identifying shape classes.

5. Main Contribution • An algorithmic framework based on symbolic shape descriptors that are robust to deformations as opposed to numeric descriptors that are often tied to specific shapes.

6. What Kind Of Deformations? Neurocranium Normal Mandibles Abnormal Normal Toy animals Abnormal 3-D Faces Shape classes: significant amount of intra-class variability

7. Deformed Infants’ Skulls Bicoronal Synostosis Sagittal Synostosis Normal Metopic Fused Sutures Coronal Sagittal Occurs when sutures of the cranium fuse prematurely (synostosis).

8. More Craniofacial Deformations Bicoronal Synostosis Metopic Synostosis Unicoronal Synostosis Sagittal Synostosis Facial Asymmetry

9. Alignment-verification • Find correspondences using numeric • signature information. • Estimate candidate transformations. Objects Database 3-D Range Scene Recognized Models Models • Verification process • selects the transformation • that produces the best • alignment.

10. Related Literature (1) • This approach has been used very successfully in industrial machine vision. Relevant investigations that use numeric signature representations for matching include: • Splash representation – Stein and Medioni (IEEE PAMI, 1992) • Spin image representation – Johnson and Hebert (IEEE PAMI, 1999).

11. Related Literature (2) • Spherical signatures – Ruiz-Correa et al. ( IEEE CVPR 2001). • Shape distributions – Osada et al. (SMI, 2001,2002). • Reflective symmetry descriptors – Kazhdan et al. (Algorithmica 2003).

12. Alignment-VerificationLimitations The approach does not extend well to the problem of identifying classes of similar shapes. In general: • Numeric shape representations are not robust to deformations. • There are not exact correspondences between model and scene. • Objects in a shape class do not align.

13. Component-Based Methodology 1 Numeric Signatures Overcomes the limitations of the alignment-verification approach define 2 Components 4 Describe spatial configuration Architecture of Classifiers Recognition And Classification Of Deformable Shapes + 3 Symbolic Signatures

14. Developed spherical spin image representation (SSI) : computational complexity O(ms). Standard spin image (SI): computational complexityO(nms) (n~103, m~104, s~103). Developed compressed SSI representation that requires O(mk) floats, k~40. Standard SIC (PCA) algorithm also requires O(md) floats but the proportionality constant is ~10 bigger. Efficient Object Recognition (1)

15. Efficient Object Recognition (3) - Clutter - Occlusion 3-D Models

16. Efficient Object Recognition (2)

17. Outline • Mathematical Background. • Formalize recognition and classification problems. • Approach and implementation. • Experimental validation: recognition and classification experiments. • Discuss future work. • Conclude.

18. SVMs - Geometry (1) Input Space X X X X X X X X X X X X X X X X X X X X X X X X X X Separate classes using a hyperplane

19. SVMs (2) Feature Space Separating Hyperplane X X X X X X X X X X X X X X X X X X X X X X X X X X Plane is a function of the inner product of the points in feature space

20. SVMs (3) F( ) = F( X ) = X Input Space Feature Space

21. SVMs – Kernel Trick (4) F( Y ) = Z F( Y’ ) = Z’ Input Space Feature Space <F( Y ), F( Y )> = <Z,Z’> = K(Y,Y’) K, kernel function

22. SVMs – Kernel Trick (5) <F( Y ), F( Y )> = <Z,Z’> = K(Y,Y’) K, kernel function example: K(Y,Y’) = exp(-g2||Y-Y’||2)

23. SVMs (2) Feature Space m m X X X X X X X X X X X X X X X X X X X X X X X X X X SVMs successful even in non-separable cases

24. Recognition Problem (1) • We are given a set of surface meshes {C1,C2,…,Cn} which are random samples of two shape classes C … C1 C2 Ck Cn …

25. Recognition Problem (2) • The problem is to use the given meshes and labels to construct an algorithm that determines whether shape class members are present in a single view range scene.

26. Classification Problem (1) • We are given a set of surface meshes {C1,C2,…,Cn} which are random samples of two shape classes C+1 and C-1, • where each surface mesh is labeled either by +1 or -1. Normal Skulls C+1 Abnormal Skulls C-1 +1 +1 +1 -1 -1 -1 -1 +1 … … C1 C2 Ck Ck+1 Ck+2 Cn

27. Classification Problem (2) • The problem is to use the given meshes and labels to construct an algorithm that predicts the label of a new surface mesh Cnew. Is this skull normal (+1) or abnormal (-1)? Cnew

28. Classification Problem (3) • We also consider the case of “missing” information: Shape class of normal heads (+1) Shape class of abnormal heads (-1) 3-D Range Scene Single View Are these heads normal or abnormal? Clutter and Occlusion

29. Assumptions • All shapes are represented as oriented surface meshes of fixed resolution. • The vertices of the meshes in the training set are in full correspondence. • Finding full correspondences : hard problem yes … but it is approachable ( use morphable models technique: Blantz and Vetter, SIGGRAPH 99; C. R. Shelton, IJCV, 2000; Allen et al., SIGGRAPH 2003).

30. + Four Key Elements To Our Approach Numeric Signatures 1 Components 2 4 Architecture of Classifiers Recognition And Classification Of Deformable Shapes Symbolic Signatures 3

31. + Numeric Signatures Numeric Signatures 1 Encode Local Surface Geometry of an Object Components 2 4 Architecture of Classifiers Symbolic Signatures 3

32. Numeric Signatures: Spin Images • Rich set of surface shape descriptors. • Their spatial scale can be modified to include local and non-local surface features. • Representation is robust to scene clutter and occlusions. 3-D faces P Spin images for point P

33. 4 + Architecture of Classifiers Components Numeric Signatures 1 define Components 2 Equivalent Numeric Signatures: Encode Local Geometry of a Shape Class Symbolic Signatures 3

34. … How To Extract Shape Class Components? Training Set Select Seed Points Compute Numeric Signatures Region Growing Algorithm Component Detector … Grown components around seeds

35. Component Extraction Example Labeled Surface Mesh Selected 8 seed points by hand Region Growing Detected components on a training sample Grow one region at the time (get one detector per component)

36. How To Combine Component Information? … 1 2 2 1 1 1 2 2 2 2 2 4 3 5 6 7 8 Extracted components on test samples Note: Numeric signatures are invariant to mirror symmetry; our approach preserves such an invariance.

37. 4 + Architecture of Classifiers Symbolic Signatures Numeric Signatures 1 Components 2 Symbolic Signatures 3 Encode Geometrical Relationships Among Components

38. 3 4 5 8 7 6 Symbolic Signature Labeled Surface Mesh Symbolic Signature at P Critical Point P Encode Geometric Configuration Matrix storing component labels

39. P 3 4 5 b a 8 7 6 Symbolic Signature Construction Normal Project labels to tangent plane at P Critical Point P tangent plane P Coordinate system defined up to a rotation b a

40. Symbolic Signatures Are Robust To Deformations P 4 3 3 4 3 3 4 3 4 4 5 5 5 5 5 8 8 8 8 8 6 7 6 7 6 7 6 7 6 7 Relative position of components is stable across deformations: experimental evidence

41. 4 + Architecture of Classifiers Architecture of Classifiers Numeric Signatures 1 Learns Components And Their Geometric Relationships Components 2 Symbolic Signatures 3

42. Proposed Architecture(Classification Example) Verify spatial configuration of the components Identify Components Identify Symbolic Signatures Class Label Input Labeled Mesh -1 (Abnormal) Two classification stages Surface Mesh

43. At Classification Time (1) Labeled Surface Mesh Surface Mesh Multi-way classifier Bank of Component Detectors Assigns Component Labels Identify Components

44. At Classification Time (2) Labeled Surface Mesh +1 Symbolic pattern for components 1,2,4 1 4 Bank of Symbolic Signatures Detectors Assigns Symbolic Labels 2 5 Two detectors 6 Symbolic pattern for components 5,6,8 8 -1

45. Finding Critical Points On Test Samples +1 Critical Point -1 1 Margin associated with the component detector classifiers Confidence Level 0

46. Architecture Implementation • ALL our classifiers are (off-the-shelf) ν-Support Vector Machines (ν-SVMs) (Schölkopf et al., 2000 and 2001). • Component (and symbolic signature) detectors are one-class classifiers. • Component label assignment: performed with a multi-way classifier that uses pairwise classification scheme. • Gaussian kernel.

47. Experimental Validation Recognition Tasks: 4 (T1 - T4) Classification Tasks: 3 (T5 – T7) No. Experiments: 5470 Setup Rotary Table Laser Recognition Classification

48. Shape Classes

49. Originals Morphs Enlarging Training Sets Using Virtual Samples Displacement Vectors Morphs Twist (5deg) + Taper - Push + Spherify (10%) Original Push +Twist (10 deg) +Scale (1.2) (14) Global Morphing Operators Physical Modeling Electrical Engineering University of Washington

50. Other Approaches • Tried standard alignment-verification. • Alignment-verification with PCA. • However, no systematic comparison was performed due to poor performance. • Existing methods for classifying shapes do not use range data. P. Golland, NIPS 2001, J. Matrin et al. IEEE PAMI 1998.