420 likes | 497 Views
Learning Decompositional Shape Models from Examples. Alex Levinshtein Cristian Sminchisescu Sven Dickinson. The Evolution of Object Recognition. Appearance-based models. Automatically built appearance-based model from video sequence
E N D
Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson
Appearance-based models Automatically built appearance-based model from video sequence (Ramanan, D. and Forsyth, D.A., “Using Temporal Coherence to Build Models of Animals”, ICCV, 2003)
Appearance-based models Constellation model (Fergus, R., Perona, P., and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, CVPR, 2003)
Hierarchical Models Manually built hierarchical model proposed by Marr And Nishihara (“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)
Our goal Automatically construct a generic hierarchical shape model from exemplars • Challenges: • Cannot assume similar appearance among different exemplars • Generic features are highly ambiguous • Generic features may not be in one-to-one correspondence
Automatically constructed Hierarchical Models Input: Question: What is it? Output:
Stages of the system Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Blob Graph Construction Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Blob Graph Construction • On the Representation and Matching of Qualitative Shape at Multiple Scales • A. Shokoufandeh, S. Dickinson, C. Jonsson, L. Bretzner, and T. Lindeberg,ECCV 2002 • Choose the largest connected component.
Blob Graph Construction Perceptual grouping of blobs: Connectivity measure: max{d1/major(A), d2/major(B)}
Blob Graph Construction Edge weights between connected blobs: • Edge weights between disconnected blobs are computed based on shortest path distances. Edge weights are invariant to articulation.
Feature matching Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Feature matching One-to-one matching. Rely on shape and context, not appearance! Many-to-many matching ?
Feature embedding Spectral embedding
Returning to our set of inputs • Many-to-many matching of every pair of exemplars.
Part Extraction Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Extracting attachment relations Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Extracting attachment relations Right arm is typically connected to torso in exemplar images !
Extracting attachment relations Number of times blobs drawn from the two clusters were attached Number of times blobs from the two clusters co-appeared in an image.
Extracting decomposition relations Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Extracting decomposition relations Sum of all flows between blobs in two clusters Number of flows between blobs in two clusters
Model construction stage summary Model Construction: • Clustering blobs into parts based on one-to-one matching results. • Recovering relations between parts based on individual matching and attachment results.
Assemble Final Model Exemplar images Extract Blob Graphs Blob graphs Match Blob Graphs (many-to-many) Many-to-many correspondences Extract Parts Model parts Extract Decomposition Relations Extract Attachment Relations Model decomposition relations Model attachment relations Assemble Final Model
Conclusions • General framework for constructing a generic decompositional model from different exemplars with dissimilar appearance. • Recovering decompositional relations requires solving the difficult many-to-many graph matching problem. • Preliminary results indicate good model recovery from noisy features.
Future work • Construct models for objects other than humans. • Provide scale invariance during matching. • Automatically learn perceptual grouping relations from labeled examples. • Develop indexing and matching framework for decompositional models.