Interactively Co- segmentating Topically Related Images with Intelligent Scribble Guidance

1. Interactively Co-segmentating Topically Related Imageswith Intelligent Scribble Guidance DhruvBatra, Carnegie Mellon University AdarshKowdle, Cornell University Devi Parikh, Toyota Technological Institute JieboLuo, Eastman Kodak Company Tsuhan Chen, Cornell University

2. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

3. Introduction • We develop an algorithm that allows users to decide what foreground is, and then guide the output of the co-segmentation algorithm towards it via scribbles

4. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

5. Energy Minimization • 1. Data (Unary) Term : indicating the cost of assigning a superpixelto foreground and backgroundclasses • 2. Smoothness (Pairwise) Term : used for penalizing label disagreement between neighbours I (·) : an indicator function that is 1(0) if the input argument is true (false) dij : the distance between features at superpixelsi and j β : a scale parameter

6. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles 3.1 Uncertainty-Based Cues 3.2 Scribble-Based Cues 3.3 Image-Level Cues 3.4 Combined Recommendation Map • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

7. Image cues: Segment Size CodewordDistribution Recommendation Map

8. 3.1 Uncertainty-Based Cues • 1. Node Uncertainty (NU):Fitting A1 = {GMMf,GMMb} to the labelledsuperpixel features. Using this learnt A1, for each superpixel we normalize the foreground and background likelihoods to get a 2-class distribution and then compute the entropy of this distribution. • 2. Edge Uncertainty (EU):To feed unlabelleddata- points to a set of classifiers and request label for the datapoint with maximal disagreement among classifier outcomes.

9. 3.1 Uncertainty-Based Cues • 3. Graph-Cut Uncertainty(GC): Capture the confidence in the energy minimizing state returned by graph cuts. 3.2 Scribble-Based Cues • 4. Distance Transform over Scribbles (DT): Compute the distance of every pixel to the nearest scribble location.

10. 3.2 Scribble-Based Cues • 5. Intervening Contours over Scribbles (IC): The value of this cue at each pixel is the maximum edge magnitude in the straight line to the closest scribble.

11. 3.3 Image-Level Cues • 6. Segment Size (SS): When very few scribbles are marked, energy minimization methods typically overs-mooth and results in “whitewash” segmentations (entire image labelled as foreground or background). • 7. Codeword Distribution over Images (CD):Motivation being that scribbling on images containing more diversity among features would lead to better foreground /background models. To compute this cue, we cluster the features computed from all superpixels in the group to form a codebook.

12. 3.4 Combined Recommendation Map • Learninga mapping F :φi →ϵi, • φi is the 7-dimensionalfeaturevector for superpixel i • ϵi is the error indicator vector • which is 1 if the predicted segmentation at node ϵi is incorrect, and 0 otherwise. • We chose logistic regression as the form of this mapping.

13. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

14. Publicly available CMU-Cornell iCoseg: 38 groups 643 images ~17 im/gp Sport

15. The CMU-Cornell iCoseg Dataset • Dataset Annotation: The ground-truth annotations for the dataset were manually generated by a single annotator using a labelling tool.

16. Dataset Statistics • Size • The histogram of the number of images in groups

17. Dataset Statistics • Appearance

18. Dataset Statistics • Scale

19. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments 5.1 Machine Experiments 5.2 User Study • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

20. 5.1 Machine Experiments

21. 5.2 User Study

22. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

23. Interactive 3D modelling

24. 3D Modeling

25. Conclusions • iCosegthat co-segments all images in the group using an energy minimization framework, and an automatic recommendation system that intelligently recommends a region among all images in the group where the user should scribble next. Achieve good quality segmentations with significantly lower time and effortthan exhaustively examining all cutouts.