O BJ C UT

1. UNIVERSITY OF OXFORD OBJ CUT M. Pawan Kumar Philip Torr Andrew Zisserman

2. Aim • Given an image, to segment the object Object Category Model Segmentation Cow Image Segmented Cow • Segmentation should (ideally) be • shaped like the object e.g. cow-like • obtained efficiently in an unsupervised manner • able to handle self-occlusion

3. Challenges Intra-Class Shape Variability Intra-Class Appearance Variability Self Occlusion

4. Motivation Magic Wand • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Object Seed Pixels Cow Image

5. Motivation Magic Wand • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Object Seed Pixels Background Seed Pixels Cow Image

6. Motivation Magic Wand • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Segmented Image

7. Motivation Magic Wand • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Object Seed Pixels Background Seed Pixels Cow Image

8. Motivation Magic Wand • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Segmented Image

9. Motivation • Problem • Manually intensive • Segmentation is not guaranteed to be ‘object-like’ Non Object-like Segmentation

10. Our Method • Combine object detection with segmentation • Borenstein and Ullman, ECCV ’02 • Leibe and Schiele, BMVC ’03 • Incorporate global shape priors in MRF • Detection provides • Object Localization • Global shape priors • Automatically segments the object • Note our method completely generic • Applicable to any object category model

11. Outline • Problem Formulation • Form of Shape Prior • Optimization • Results

12. Problem • Labelling m over the set of pixels D • Shape prior provided by parameter Θ • Energy E (m,Θ) = ∑Φx(D|mx)+Φx(mx|Θ) + ∑ Ψxy(mx,my)+ Φ(D|mx,my) • Unary terms • Likelihood based on colour • Unary potential based on distance from Θ • Pairwise terms • Prior • Contrast term • Find best labelling m* = arg min ∑ wi E (m,Θi) • wi is the weight for sample Θi Unary terms Pairwise terms

13. MRF • Probability for a labellingconsists of • Likelihood • Unary potential based on colour of pixel • Prior which favours same labels for neighbours (pairwise potentials) mx m(labels) Prior Ψxy(mx,my) my Unary Potential Φx(D|mx) x y D(pixels) Image Plane

14. Example Cow Image Object Seed Pixels Background Seed Pixels Φx(D|obj) x … x … Φx(D|bkg) Ψxy(mx,my) y … y … … … … … Prior Likelihood Ratio (Colour)

15. Example Cow Image Object Seed Pixels Background Seed Pixels Prior Likelihood Ratio (Colour)

16. Contrast-Dependent MRF • Probability of labelling in addition has • Contrast term which favours boundaries to lie on image edges mx m(labels) my x Contrast Term Φ(D|mx,my) y D(pixels) Image Plane

17. Example Cow Image Object Seed Pixels Background Seed Pixels Φx(D|obj) x … x … Φx(D|bkg) Ψxy(mx,my)+ Φ(D|mx,my) y … y … … … … … Prior + Contrast Likelihood Ratio (Colour)

18. Example Cow Image Object Seed Pixels Background Seed Pixels Prior + Contrast Likelihood Ratio (Colour)

19. Our Model • Probability of labelling in addition has • Unary potential which depend on distance from Θ (shape parameter) Θ (shape parameter) Unary Potential Φx(mx|Θ) mx m(labels) my Object Category Specific MRF x y D(pixels) Image Plane

20. Example Cow Image Object Seed Pixels Background Seed Pixels ShapePriorΘ Prior + Contrast Distance from Θ

21. Example Cow Image Object Seed Pixels Background Seed Pixels ShapePriorΘ Prior + Contrast Likelihood + Distance from Θ

22. Example Cow Image Object Seed Pixels Background Seed Pixels ShapePriorΘ Prior + Contrast Likelihood + Distance from Θ

23. Outline • Problem Formulation • E (m,Θ) = ∑Φx(D|mx)+Φx(mx|Θ) + ∑ Ψxy(mx,my)+ Φ(D|mx,my) • Form of Shape Prior • Optimization • Results

24. Layered Pictorial Structures (LPS) • Generative model • Composition of parts + spatial layout Layer 2 Spatial Layout (Pairwise Configuration) Layer 1 Parts in Layer 2 can occlude parts in Layer 1

25. Layered Pictorial Structures (LPS) Cow Instance Layer 2 Transformations Θ1 P(Θ1) = 0.9 Layer 1

26. Layered Pictorial Structures (LPS) Cow Instance Layer 2 Transformations Θ2 P(Θ2) = 0.8 Layer 1

27. Layered Pictorial Structures (LPS) Unlikely Instance Layer 2 Transformations Θ3 P(Θ3) = 0.01 Layer 1

28. LPS for Detection • Learning • Learnt automatically using a set of examples • Detection • Matches LPS to image using Loopy Belief Propagation • Localizes object parts

29. Outline • Problem Formulation • Form of Shape Prior • Optimization • Results

30. Optimization • Given image D, find best labelling as m* = arg max p(m|D) • Treat LPS parameter Θas a latent (hidden) variable • EM framework • E : sample the distribution over Θ • M : obtain the labelling m

31. E-Step • Given initial labelling m’, determine p(Θ|m’,D) • Problem Efficiently sampling from p(Θ|m’,D) • Solution • We develop efficient sum-product Loopy Belief Propagation (LBP) for matching LPS. • Similar to efficient max-product LBP for MAP estimate • Felzenszwalb and Huttenlocher, CVPR ‘04

32. Results • Different samples localize different parts well. • We cannot use only the MAP estimate of the LPS.

33. M-Step • Given samples from p(Θ|m’,D), get new labelling mnew • Sample Θiprovides • Object localization to learn RGB distributions of object and background • Shape prior for segmentation • Problem • Maximize expected log likelihood using all samples • To efficiently obtain the new labelling

34. M-Step w1 = P(Θ1|m’,D) Cow Image Shape Θ1 RGB Histogram for Background RGB Histogram for Object

35. M-Step w1 = P(Θ1|m’,D) Cow Image Shape Θ1 Θ1 m(labels) Image Plane D(pixels) • Best labelling found efficiently using a Single Graph Cut

36. Segmentation using Graph Cuts Obj Cut Φx(D|bkg) + Φx(bkg|Θ) x … • Ψxy(mx,my)+ • Φ(D|mx,my) y … … … m z … … Φz(D|obj) + Φz(obj|Θ) Bkg

37. Segmentation using Graph Cuts Obj x … y … … … m z … … Bkg

38. M-Step w2 = P(Θ2|m’,D) Cow Image Shape Θ2 RGB Histogram for Background RGB Histogram for Object

39. M-Step w2 = P(Θ2|m’,D) Cow Image Shape Θ2 Θ2 m(labels) Image Plane D(pixels) • Best labelling found efficiently using a Single Graph Cut

40. M-Step Θ1 Θ2 w1 + w2 + …. Image Plane Image Plane m* = arg min ∑ wi E (m,Θi) • Best labelling found efficiently using a Single Graph Cut

41. Outline • Problem Formulation • Form of Shape Prior • Optimization • Results

42. Results Using LPS Model for Cow Image Segmentation

43. Results Using LPS Model for Cow In the absence of a clear boundary between object and background Image Segmentation

44. Results Using LPS Model for Cow Image Segmentation

45. Results Using LPS Model for Cow Image Segmentation

46. Results Using LPS Model for Horse Image Segmentation

47. Results Using LPS Model for Horse Image Segmentation

48. Results Image Our Method Leibe and Schiele

49. Results Shape Shape+Appearance Appearance Without Φx(mx|Θ) Without Φx(D|mx)

50. Conclusions • New model for introducing global shape prior in MRF • Method of combining detection and segmentation • Efficient LBP for detecting articulated objects • Future Work • Other shape parameters need to be explored • Method needs to be extended to handle multiple visual aspects