Learning Decompositional Shape Models from Examples

1. Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson University of Toronto

2. Hierarchical Models Manually built hierarchical model proposed by Marr And Nishihara (“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)

3. Our goal Automatically construct a generic hierarchical shape model from exemplars • Challenges: • Cannot assume similar appearance among different exemplars • Generic features are highly ambiguous • Generic features may not be in one-to-one correspondence

4. Layered Motion SegmentationsKumar, Torr and Zisserman, ICCV 2005 • Models image projection, lighting and motion blur • Models spatial continuity, occlusions, and works over multiple frames (cf. earlier work by Jojic & Frey, CVPR 2001) • Estimates the number of segments, their mattes, layer assignment, appearance, lighting and transformation parameters for each segment • Initialization using loopy BP, refinement using graph cuts

5. Constellation models Fergus, R., Perona, P., and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, CVPR 2003

6. Categorical features Match

7. Constructing a Hierarchical Model from Examples Input: Question: What is it? Output:

8. Overview of the Approach Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

9. Blob Graph Construction Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

10. Blob Graph Construction • The Representation and Matching of Categorical Shape • A. Shokoufandeh, L. Bretzner, D. Macrini, M.F. Demirci, C. Jonsson, and S. Dickinson, CVIU, Vol. 103, 2006, pp 139--154 • Edges are invariant to articulation • Choose the largest connected component.

11. Blob Graph Construction Perceptual grouping of blobs: Connectivity measure: max{d1/major(A), d2/major(B)}

12. Feature matching Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

13. Feature matching One-to-one matching. Rely on shape and context, not appearance! Many-to-many matching

14. A Many-to-Many Graph Matching Framework 1. Embed graphs with low distortion to yield weighted point distributions. 2. Compute many-to-many correspondences between the two distributions using EMD. 3. The computed flows yield a many-to-many node correspondence between the two graphs. Demirci, Shokoufandeh, Keselman, Bretzner, and Dickinson (IJCV 2006)

15. Feature embedding and EMD Spectral embedding

16. Returning to our set of inputs • Many-to-many matching of every pair of exemplars.

17. Part Extraction Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

18. Many-to-many matching results

19. Extracting parts • Part – a collection of blobs. • Ideal part • Represents blobs that occur frequently and participate in one-to-one correspondence across many exemplars. • Finding parts • From the pairwise matching results, find clusters (cliques) of blobs matching one-to-one.

21. Results of the part extraction stage

22. Extracting attachment relations Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

23. Extracting attachment relations Number of times blobs drawn from the two clusters were attached is high Right arm is typically connected to torso in exemplar images ! Number of times blobs from the two clusters co-appeared in an image. Torso Right Arm

24. Extracting attachment relations Number of times blobs drawn from the two clusters were attached Number of times blobs from the two clusters co-appeared in an image. Threshold PA to get part attachment

25. Extracting decomposition relations Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

26. Extracting decomposition relations Left Arm Upper Lower

27. Extracting decomposition relations Sum of all flows between blobs in two clusters Number of flows between blobs in two clusters Combine PA and PF to obtain a decomposition score

28. Model construction stage summary Model Construction: • Clustering blobs into parts based on one-to-one matching results. • Recovering relations between parts based on individual matching and attachment results.

29. Assemble Final Model Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Extract Decomposition Relations Extract Attachment Relations Model parts Model decomposition relations Model attachment relations Assemble Final Model

30. Results

32. Experiments

33. List of system parameters

40. Conclusions • Generic models must be defined at multiple levels of abstraction, as Marr proposed. • Coarse shape features, such as blobs, are highly ambiguous and cannot be matched without contextual constraints. • Moreover, features that exist at different levels of abstraction must be matched many-to-many in the presence of noise. • The many-to-many matching results can be analyzed to yield both the parts and relations of a decompositional model. • Preliminary results indicate that a limited decompositional model can be learned from a set of noisy examples.

41. Future work • Construct models for objects other than humans – objects with richer decompositional hierarchies. • Automatically learn perceptual grouping relations between blobs from labeled examples. • Develop indexing and matching frameworks for decompositional models.