Recursive Composition in Computer Vision

1. Recursive Composition in Computer Vision Leo Zhu CSAIL MIT Joint work with Chen, Yuille, Freeman and Torralba

2. Ideas behind Recursive Composition Pattern Theory. Grenander 94 Compositionality. Geman 02, 06 Stochastic Grammar. Zhu and Mumford 06 • How to deal with image complexity • A general framework for different vision tasks • Rich representation and tractable computation

3. Recursive Composition • Representation • Recursive Compositional Models (RCMs) • Inference • Recursive Optimization • Learning • Supervised Parameter Estimation • Unsupervised Recursive Dictionary Learning • RCM-1: Deformable Object • RCM-2: Articulated Object • RCM-3: Scene (Entire Image)

4. Model Deformable Object • Flat MRF • Nodes: object parts • Edges: spatial relations • Limitations: • Short range interaction • Sparse

5. Recursive Composision

6. Recursive Compositional Models:RCM-1 x: image y: (position, scale, orientation) graph=(nodes, edges) a: index of node b: child of a f: appearances on node a g: potentials on edges (a,b)

7. RCM-1: the Recursive Formula Recursion x: image ; y: (position, scale, orientation); Vertical independency; Self-similarity;

8. Recursive Composition • Representation • Recursive Compositional Models (RCMs) • Inference • Recursive Optimization • Learning • Supervised Parameter Estimation • Unsupervised Recursive Dictionary Learning

9. Polynomial-time Inference Recursion • Polynomial-time Complexity: Inference task: Recursive Optimization:

10. Supervised Learning Collins 02. Taskar et al. 04 • Supervised learning • Perceptron algorithm (MLE, max margin – svm) • Parameter estimation needs fast inference.

11. Supervised learning by Perceptron Algorithm where Inference is critical for learning • Goal: • Input: a set of training images with ground truth . Initialize parameter vector. • Training algorithm (Collins 02): Loop over training samples: i = 1 to N Step 1: find the best using inference: Step 2: Update the parameters: End of Loop.

12. Recursive Composition • Representation • Recursive Compositional Models (RCMs) • Inference • Recursive Optimization (Polynomial-time) • Learning • Supervised Parameter Estimation • RCM-1: Deformable Object

13. RCM-1: Multi-level Potentials = * [ Gabor, Edge, …] Potentials for appearance

14. RCM-1: Multi-level Potentials (position, scale, orientation) • Potentials for shape: triplet descriptors

15. The Inference Results after Supervised Learning

17. Segmentation Results

18. Evaluations: Segmentation and Parsing • Segmentation (Accuracy of pixel labeling) • The proportion of the correct pixel labels (object or non-object) • Parsing (Average Position Error of matching) • The average distance between the positions of leaf nodes of the ground truth and those estimated in the parse tree

19. Recursive Composition • Modeling: (Representation) • Recursive Compositional Models (RCMs) • Inference: (Computing) • Recursive Optimization (Polynomial-time) • Learning: • Supervised Parameter Estimation • Unsupervised Recursive Learning • RCM-1: deformable object

20. Unsupervised Learning Correspondence is unknown ? Combinatorial Explosion problem • Task: given 10 training images, no labeling, no alignment, highly ambiguous features. • Induce the structure (nodes and edges) • Estimate the parameters.

21. Recursive Dictionary Learning Recursion Barlow 94. • Multi-level dictionary (layer-wise greedy) • Bottom-Up and Top-Down recursive procedure • Three Principles: • Recursive Composition • Suspicious Coincidence • Competitive Exclusion

22. 10 images for training

23. Bottom-up Learning Suspicious Coincidence Clustering Composition Competitive Exclusion

24. The Dictionary: From Generic Parts to Object Structures Unified representation (RCMs) and learning Bridge the gap between the generic features and specific object structures

25. Dictionary Size, Part Sharing and Computational Complexity More Sharing

29. Top-down refinement Fill in missing parts Examine every node from top to bottom

31. Evaluations of Unsupervised Learning

32. Scale up the System: Issue I More classes/viewpoints -> more training/detection cost

33. Scale up the System: Issue II No enough data for rare viewpoints/classes

34. Our Strategy Joint multi-class multi-view learning Appearance sharing Part sharing

35. Joint Multi-Class Multi-View Learning 120 templates: 5 viewpoints & 26 classes

36. Different Viewpoints Share same appearance

37. Different Classes Share Common Parts

38. Compact Hierarchical Dictionary

39. Dense Part Sharing at Low Levels: Layer-2

40. Less Part Sharing: Layer-3

41. Sparse Part Sharing at High Levels: Layer-4

42. Re-usable Parts: All Layers

43. The more classes/viewpoints, the more amount of part sharing

44. Multi-View Single Class Performance

45. Recursive Composition • Representation • Recursive Compositional Models (RCMs) • Inference • Recursive Optimization (Polynomial-time) • Learning • Supervised Parameter Estimation • RCM-1: Deformable Object • RCM-2: Articulated Object

46. RCM-2 for Articulated Object: Horses multiple poses y=(switch, position, scale, orientation) Composition Switch

47. RCM-2 for Human Body

49. Recursive Composition • Representation • Recursive Compositional Models (RCMs) • Inference • Recursive Optimization (Polynomial-time) • Learning • Supervised Parameter Estimation • RCM-1: Deformable Object • RCM-2: Articulated Object • RCM-3: Scene (Entire Image)

50. Image Scene Parsing Task: Image Segmentation andLabeling