Silhouette-based Object Phenotype Recognition using 3D Shape Priors

1. Silhouette-based Object Phenotype Recognition using 3D Shape Priors Yu Chen1 Tae-Kyun Kim2 Roberto Cipolla1 University of Cambridge, Cambridge, UK1 Imperial College, London, UK2

2. Problem Description • Task: To identify the phenotype class of deformable objects. • Given a gallery of canonical-posed silhouettes in different phenotype classes. • Can we find out ?

3. Problem Description • Motivation: • Pose recognition is widely investigated; • Phenotype recognition is somehow overlooked; • Applications? • Difficulty: • Pose and camera viewpoint variations are more dominant than the phenotype variation.

4. Problem Description • 2D approaches hardly work in this case. • Our strategy: make use of the 3D shape prior of deformable objects. • Shall we use a purely generative approach? • No! Too expensive to perform for a recognition task!

5. Solution: Two-Stage Model • Main Ideas: Discriminative + Generative • Two stages: • Hypothesising • Discriminative; • Using random forests; • Shape Synthesis and Verification • Generative; • Synthesising 3D shapes using shape priors; • Silhouette verification. • Recognition by a model selection process.

6. Parameter Hypothesizing • Use 3 RFs to quickly hypothesize phenotype, pose, and camera parameters. • Learned on synthetic silhouettes generated by the shape priors. FS: Phenotype classifier (canonical pose) FA: Pose classifier FC: Camera pose classifier

7. Examples of Tree Classifiers The phenotype classifier The pose classifier

8. Training RF Classifiers • Random Features: • Rectangle pairs with random sizes and locations. • Difference of mean intensity values [Shotton et al. 09] • Feature error compensation for phenotype classifier; • Criteria Function: • Similarity-aware diversity index.

9. Shape Synthesis and Verification • Generate 3D shapes V • From candidate parameters given by RFs. • Use GPLVM shape priors [Chen et al.’10]. • Compare the projection of V with the query silhouette Sq. • Oriented Chamfer matching (OCM). [Stenger et al’03]

10. Experiments • Testing data: • Manually segmented silhouettes; • Current Datasets • Human jumping jack • (13 instances, 170 images); • Human walking • (16 instances, 184 images); • Shark swimming • (13 instances, 168 images). • Phenotype Categorisation

11. Comparative Approaches: • Learn a single RF phenotype classifier; • Histogram of Shape Context (HoSC) • [Agarwal and Triggs, 2006] • Inner-Distance Shape Context (IDSC) • [Ling and Jacob, 2007] • 2D Oriented Chamfer matching (OCM) • [Stenger et al. 2006] • Mixture of Experts for the shape reconstruction • [Sigal et al. 2007]. • Modified into a recognition algorithm

12. Comparative Approaches: • Internal comparisons: • Proposed method with both feature error modelling and similarity-aware criteria function (G+D); • Proposed method w.o feature error modelling (G+D–E); • Proposed method w.o similarity-aware criteria function (G+D–S) • Using standard diversity index instead.

13. Recognition Performance • Cross-validation by splitting the dataset instances. • 5 phenotype categories for every test. • Selecting one instance from each category.

14. Recognition Performance • How the parameters of RFs affect the performance? • Max Tree Depth dmax • Tree Number NT

15. Qualitative Results of SVR Left: Input image/silhouette; Centre: Using RF-hypothesizes; Right: Using the optimisation-based approach.

16. Qualitative Results of SVR

17. Take-Home Messages • Phenotype recognition is difficult but still possible; • Combing discriminative and generative cues can greatly speed up the inference; • A divide-and-conquer strategy can help improve the recognition rate.

18. Future Work • Explore the application on more complicated poses and more categories. • E.g. Boxing, gardening, other sports, etc. • Data collection; • Automate the silhouette extraction. • E.g. Kinect.

19. The End Questions?