Learning Jigsaws for clustering appearance and shape

1. Learning Jigsawsfor clustering appearance and shape Anitha Kannan, John Winn and Carsten Rother NIPS 2006

2. Learning jigsaws Aim: Cluster regions in images with similar appearance and shape. Examples of clusters (jigsaw pieces) Eye Cheek Noses Eyebrows

3. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

4. Clustering image patches Patches Clusters [Leibe & Schiele, BMVC 2003]

5. Clustering image patches Cluster? Patch includes background

6. Clustering image patches Cluster? Patch wrong shape

7. Clustering image patches Cluster? Part is occluded

8. Clustering image patches Cluster? Need to adapt the patch shape depending on the image.

9. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

10. Aims of jigsaw model Learn clusters (jigsaw pieces) so that: Clustered patches have similar shape and appearance Patches are as large as possible Every image pixel belongs to exactly one patch (i.e. the images are segmented into patches)

11. The Jigsaw model Jigsaw J Region of constant offset ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

12. The Jigsaw model Jigsaw Jigsaw J Mean μ(z) and inverse variance λ(z) for each jigsaw pixel z. Appearance model offset at pixel i Offset map prior (Potts model) Image Offset map I L cost of patch boundary

13. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

14. Toy example Image with segmentation Jigsaw Mean Variance Learned by iteratively maximising joint probability w.r.t. jigsaw and offset maps (see paper for details)

15. Comparison to epitome model [Jojic et al., ICCV 2003] Jigsaw Epitome • data-driven patch shape • non-overlapping patches • fixed patch shape • overlapping patches

16. Faces example Face images with segmentations Jigsaw 128128 mean Source: Olivetti face database

17. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

18. Learning the jigsaw pieces Jigsaw J ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

19. Learning the jigsaw pieces Jigsaw J ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

20. Learning the jigsaw pieces Jigsaw J ... Image Offset map Image Offset map Image Offset map I L I L I L 1 1 2 2 N N

21. Shape clustering on faces Commonly used pieces Jigsaw showing pieces

22. Road map • Clustering image patches • The Jigsaw model • Results on toy and real images • Learning jigsaw pieces • Discussion and conclusions

23. Jigsaw applications • Can be used as ‘plug-and-play’ replacement for fixed-shape patch model in existing systems. • Applications include: • Object recognition/detection • Object segmentation • Stereo matching • Texture synthesis • Super-resolution • Motion segmentation • Image/video compression

24. Jigsaw extensions • Allows rotation/scaling/deformationof the patches. • Incorporating shape clustering into the probabilistic model • Incorporating additional invariances e.g. to illumination • Apply to other domains: audio, biology

25. Conclusions • Jigsaw model allows learning the shape and appearance of recurring regions in images. • Jigsaw performs unsupervised discovery of object parts. • Future work: apply jigsaw to use shape as well as appearance for object recognition and other vision applications.

26. Thank you Jigsaw paper (compressed)