Learning Jigsaws for clustering appearance and shape

1. Learning Jigsawsfor clustering appearance and shape John Winn, Anitha Kannan and Carsten Rother NIPS 2006

2. Learning jigsaws Aim: Cluster regions in images with similar appearance and shape. Examples of clusters (jigsaw pieces) Eye Cheek Noses Eyebrows

3. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

4. Clustering image patches Patches Clusters [Leibe & Schiele, BMVC 2003]

5. Clustering image patches Cluster? Patch wrong shape

6. Clustering image patches Cluster? Patch wrong shape

7. Clustering image patches Cluster? Part is occluded

8. Clustering image patches Cluster? Need to adapt the patch shape depending on the image.

9. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

10. Aims of jigsaw model Learn clusters (jigsaw pieces) so that: Clustered patches have similar shape and appearance Patches are as large as possible Every image pixel belongs to exactly one patch (i.e. the images are segmented into patches)

11. The Jigsaw model Jigsaw J Region of constant offset ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

12. The Jigsaw model Jigsaw Jigsaw J Mean μ(z) and inverse variance λ(z) for each jigsaw pixel z. Appearance model offset at pixel i Offset map prior (Potts model) Image Offset map I L cost of patch boundary

13. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

14. Toy example Image with segmentation Jigsaw Mean Variance Learned by iteratively maximising joint probability w.r.t. jigsaw and offset maps (see paper for details)

15. Comparison: Mixture of Gaussians Cluster centres • fixed patch shape

16. Comparison: Epitome Epitome • fixed patch shape • translation invariant [Jojic et al., ICCV 2003]

17. Comparison: Jigsaw Jigsaw • learned patch shape • translation invariant • non-overlapping patches

18. Comparison: all methods MoG Original Error = 0.103 Jigsaw Epitome Error = 0.071 Error = 0.054

19. Faces example Face images with segmentations Jigsaw 128128 mean Source: Olivetti face database

20. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

21. Learning the jigsaw pieces Jigsaw J ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

22. Learning the jigsaw pieces Jigsaw J ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

23. Learning the jigsaw pieces Jigsaw J ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

24. Commonly used pieces Shape clustering on faces Jigsaw showing pieces

25. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

26. Jigsaw applications • Can be used as ‘plug-and-play’ replacement for fixed-shape patch model in existing systems. • Applications include: • Object recognition/detection • Object segmentation • Stereo matching • Texture synthesis • Super-resolution • Motion segmentation • Image/video compression

27. Future work • Allow rotation/scaling/deformationof the patches. • Incorporate shape clustering into the probabilistic model • Incorporate additional invariances e.g. to illumination • Apply to other domains: audio, biology

28. Conclusions • Jigsaw model allows learning the shape and appearance of recurring regions in images. • Jigsaw performs unsupervised discovery of object parts.

29. Thank you http://johnwinn.org Jigsaw paper (compressed)

31. Comparison: Epitome Epitome • fixed patch shape • translation invariant • overlapping patches [Jojic et al., ICCV 2003]

32. Patch averaging MoG Epitome Error = 0.071 Error = 0.054