Unsupervised Learning of Categorical Segments in Image Collections

1. The Sixth IEEE Computer Society Workshop on Perceptual Organization in Computer Vision (POCV 2008) Unsupervised Learning of Categorical Segments in Image Collections *California Institute of Technology **Technion Marco Andreetto*, Lihi Zelnik-Manor**, Pietro Perona*

2. Outline • Motivation and related work • A probabilistic model for single image segmentation • Unsupervised learning of categorical segments • Experimental results • Conclusions and future works

3. Outline • Motivation and related work • A probabilistic model for single image segmentation • Unsupervised learning of categorical segments • Experimental results • Conclusions and future works

4. Motivation

5. Motivation Normalized cuts: Shi and Malik PAMI 2000

6. Motivation

7. Motivation

8. Motivation Categorical segments: from human segmentation

9. Motivation

10. Related works • Russell et al. CVPR 2006 • Cao and Fei-Fei ICCV 2007 • Wang and Grimson NIPS 2007 • Andreetto et al. ICCV 2007

11. Outline • Motivation and related work • A probabilistic model for single image segmentation • Unsupervised learning of categorical segments • Experimental results • Conclusions and future works

12. An image as a set of segments

13. An image as a set of segments K = 2 N

14. An image as a set of segments K = 2  N Segment probability

15. An image as a set of segments K = 2  N Segment probability Segment density fk K

16. Image formation K = 2  Segment probability c Label Segment density fk x N K

17. Likelihood of x to be in cluster k What we’re looking for Observed Probabilistic model for clustering  c fk x N K

18. Non-parametric densities Sum of local kernels

19. Outline • Motivation and related work • A probabilistic model for single image segmentation • Unsupervised learning of categorical segments • Experimental results • Conclusions and future works

20. Learning categorical segments M  N c gk w x fk K K Segment appearance Joint for all images Segment shape/color Specific per image

21. Visual words w1 w2 VQ Filter Bank w3 … wN … • Filter bank: 17 outputs • 256 visual words Winn et al. ICCV 2005

22. M  N c gk w x fk K K Inference

23. Gibbs sampling

24. Gibbs sampling Prior term: Number of pixels in image m assigned to segment k

25. Gibbs sampling Prior term: Number of pixels in image m assigned to segment k Visual words term: Number of visual word h assigned to segments k

26. Gibbs sampling Prior term: Number of pixels in image m assigned to segment k Visual words term: Number of visual word h assigned to segments k Segment term: Affinity between observations i and j Non-parametric density Estimate for segment k

27. Outline • Motivation and related work • A probabilistic model for single image segmentation • Unsupervised learning of categorical segments • Experimental results • Conclusions and future works

28. Experimental results (MSRC)

29. Classification results (MSRC) Running time: 18.75 sec. per image

30. Experimental results (Labelme)

31. Categorical segments (Labelme) Segment 1: Foliage Segment 2: Buildings Segment 3: Street pavement Segment 1: Sky

32. Categorical segments (scenes)

33. Outline • Motivation and related work • A probabilistic model for single image segmentation • Unsupervised learning of categorical segments • Experimental results • Conclusions and future works

34. Conclusions • We presented a model for unsupervised learning of categorical segments • We describe an inference method based on Gibbs sampling • We show some experimental results on a standard dataset MSRC v1.

35. Future work • Faster inference method (variational approximation) • Automatic inference of the number of segments • Learning geometric relationships between segments

36. Thank You