LOCUS (Learning Object Classes with Unsupervised Segmentation)

1. LOCUS(Learning Object Classes with Unsupervised Segmentation) A variational approach to learning model-based segmentation. John WinnMicrosoft Research Cambridge with Nebojsa Jojic, MSR Redmond 7th July 2006

2. Overview • Learning object models • The LOCUS model • Experiments & results • Extensions to LOCUS

3. Goal Long Term Goal Recognise ~10,000 object classes.

4. Learning from ‘buckets’ of images Learningalgorithm Horsemodel • Object Segmentation • Object Recognition • Object Detection

5. + Object segmentation LOCUS Horsemodel

6. Related work

7. Constellation models • Weakly supervised • Probabilistic framework • Sparse • No segmentation Object class recognition by unsupervised scale-invariant learning. R. Fergus, P. Perona, and A. Zisserman. CVPR 2003 A Bayesian approach to unsupervised One-Shot learning of Object categories. L. Fei-Fei, R. Fergus, and P. Perona. ICCV 2003

8. Fragment-based • Dense model • Supervised • Non-probabilistic • No global shape model Learning to segment. E. Borenstein and S. Ullman. ECCV 2004 Combining top-down and bottom-up segmentation. E. Borenstein, E. Sharon, and S. Ullman. CVPR 2004

9. Codebook-based • Probabilistic • Dense model • Supervised • Ad-hoc inference Combined object categorization and segmentation with an implicit shape model. B. Leibe, A. Leonardis, and B. Schiele. ECCV ‘04

10. OBJ CUT • Probabilistic • Dense model • Supervised • Requires video

11. LOCUS overview • Weakly supervised learning Buckets of images - no annotation required. • Probabilistic generative modelof both object and background. • Dense modelAll pixels modelled, not just at interest points. • Combines global and local cuesModels global shape and local appearance + edges. • Iterative inference processSimultaneous localisation, segmentation, pose estimation.

12. The LOCUS model

13. LOCUS model Shared between images Class shape π Class edge sprite μo,σo Deformation field D Position & size T Different for each image Mask m Edge image e Background appearance λ0 Object appearance λ1 Image

14. background Background mixture coefficients Objectmixture coefficients object λ0 λ1 Shared mixture components: LOCUS model: appearance Mask m Image z

15. favours segmentation along contrast edges LOCUS model: mask background object 8-neighbour Markov Random Field (as used in GrabCut)

16. Class shapeπ Transformation TN T4 T2 T3 T1 … LOCUS model: shape/position …

17. TN T1 Iterative inference Class shapeπ Iteration #1 T2 T3 T4 … …

18. TN T1 Iterative inference Class shapeπ Iteration #2 T2 T3 T4 … …

19. TN T1 Iterative inference Class shapeπ Iteration #3 T2 T3 T4 … …

20. TN T1 Iterative inference Class shapeπ Iteration #5 T2 T3 T4 … …

21. TN T1 Iterative inference Class shapeπ Iteration #8 T2 T3 T4 … …

22. TN T1 Iterative inference Class shapeπ Iteration #12 T2 T3 T4 … …

23. Non-rigid objects Class shapeπ Translation and scale is not enough.

24. Deformation field D 5x5 blocks T Prior ensures smoothness LOCUS model: pose Class shapeπ

25. TDN TD1 TD2 TD3 … LOCUS model: pose Class shapeπ …

26. Class edge sprite μo,σo TDN TD1 TD2 TD3 … Edge images e … LOCUS model: edge Original images …

27. LOCUS model: overview Shared between images Class shape π Class edge sprite μo,σo Deformation field D Position & size T Different for each image Mask m Edge image e Background appearance λ0 Object appearance λ1 Image

28. Inference • Aim to infer all latent variables, • For each image:background appearance λ0, object appearanceλ1, deformation D, transformation T, mask m, • Class variables: shape π, edge sprite μo, σo. • Bayesian inference is carried out using variational message passing with a fully factorised variational distribution. • Optimisation of grid-structured variational free energy terms (relating to the deformation field D and the mask m) achieved using graph cuts.

29. Experiments & results

30. Experiments LOCUS applied to 8 sets of 20 images each containing objects of the same class. • Horses • Faces • Cars (rear) • Cars (side) • Motorbikes • Aeroplanes • Cows • Trees For each class, we ran separate experiments for color and texture appearance models.

31. Results: horses

32. Results: horses

33. Results: cars

34. Results: cars

35. Faces Cars (rear) Motorbikes Planes Cows Trees Results: remaining classes

36. Segmentation accuracy To evaluate segmentation quantitively, we used hand segmentations for horses and cars (side).

37. Object registration Transformation + deformation field registers object outlines (and some internal edges).

38. Object registration

39. Extensions to LOCUS

40. Recognition + segmentation Object recognition using only global shape: Overall: 88% accuracy.

41. Probabilistic Index Maps 2 indices 9 indices Each image has a ‘palette’ of appearance models – palette invariance.

42. Probabilistic Index Maps

43. Learning objects from video Object shape Object edge sprite

44. Locumotion Add flow and track constraints to achieve motion segmentation: Tracking/flow estimation by Larry Zitnick

45. Conclusions • LOCUS gives unsupervised segmentations of accuracy equivalent to state-of-the-art supervised methods. • General-purpose model allows: • Object localisation • Pose estimation • Object segmentation • Motion segmentation/object tracking • Object recognition/detection (in combination with discriminative model)

46. Questions ?