From Interactive to Semantic Image Segmentation

1. From Interactive to SemanticImage Segmentation VarunGulshan Supervisors: Prof. Andrew Blake Prof. Andrew Zisserman 20 Jan 2012

2. Two segmentation tasks sky background building tree tree object person car car road bench Interactive segmentation Semantic segmentation

3. Thesis Flow Chapter 4: Star convexity Chapter 3: Texture Features Chapter 6: Superpixel based classification Chapter 5: Segmenting humans Low level cues Mid level cues Bounding box interaction + Top down cues Fully automatic segmentation Interactive segmentation Semantic segmentation

4. Chapter 3: Features for interactive segmentation Low level texture features for improving interactive segmentation methods.

5. Texture features Pure Texture Feature (L-shape): Texture + Gray Feature (L-shape):

6. Texture features Pure Texture Feature (Plus-shape): Texture + Gray Feature (Plus-shape):

7. Camouflage Image Dataset Introduced a dataset of 50 Camouflage images, to demonstrate the power of texture features.

8. Quantitative evaluation Gray +21% +14% RGB +4% +7% Huge gain in accuracy obtained using texture features on top of gray scale images. Significant improvement on top of RGB images.

9. Chapter 4: Star Convexity and Extensions Mid level shape constraints for reducing user effort in interactive segmentation systems.

10. Chapter 4: Star convexity Single Star Multiple Stars Geodesic Star

11. Robot user evaluation False negative False positive Initial brush strokes Error segmentation Segmentation output with current interaction New Brush Stroke New Brush Stroke New brush stroke placed Centre of connected component Biggest connected component Process is repeated upto 20 strokes Segmentation after 20 strokes Updated segmentation

12. Robot user evaluation Our method takes least effort

13. Chapter 5: Learning to segment humans Using top down cues to segment specific object categories.

14. Segmenting humans Bounding box (given/detected) Top down HOG prediction Bottom up refinement

15. Kinect Data Acquisition RGB image Kinect scene labels Cleaned up Ground truth Dataset of roughly 3500 images acquired using the Kinect

16. Top down learning Local Image Local HOG Local mask Classifier trained to predict segmentation masks for local windows based on their HOG descriptor.

17. Bottom up refinement ….. Top down segmentation Local Color model window Local color model unaries Final segmentation

18. Chapter 6: Semantic segmentation Fully automatic segmentation based upon learning from multiple superpixelisations.

19. Combing multiple superpixelisations GlobalPb Veksler QuickShift Various methods to learn from multiple superpixelisations explored: 1. Avg-Indep 2. Avg-Union 3. LPβ-Indep 4. IofR-Joint

20. Quantitative evaluation Single superpixelisation +7% +5% +6% +3% Multiple superpixelisations Combining multiple superpixelisations improves performance.

21. Novel pairwise features .... .... CRF trained jointly for appearance and novel pairwise features.

22. Over to you!