1 / 25

Interactively Co- segmentating Topically Related Images with Intelligent Scribble Guidance

Interactively Co- segmentating Topically Related Images with Intelligent Scribble Guidance. Dhruv Batra , Carnegie Mellon University Adarsh Kowdle , Cornell University Devi Parikh, Toyota Technological Institute Jiebo Luo , Eastman Kodak Company Tsuhan Chen, Cornell University.

cade
Download Presentation

Interactively Co- segmentating Topically Related Images with Intelligent Scribble Guidance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Interactively Co-segmentating Topically Related Imageswith Intelligent Scribble Guidance DhruvBatra, Carnegie Mellon University AdarshKowdle, Cornell University Devi Parikh, Toyota Technological Institute JieboLuo, Eastman Kodak Company Tsuhan Chen, Cornell University

  2. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

  3. Introduction • We develop an algorithm that allows users to decide what foreground is, and then guide the output of the co-segmentation algorithm towards it via scribbles

  4. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

  5. Energy Minimization • 1. Data (Unary) Term : indicating the cost of assigning a superpixelto foreground and backgroundclasses • 2. Smoothness (Pairwise) Term : used for penalizing label disagreement between neighbours I (·) : an indicator function that is 1(0) if the input argument is true (false) dij : the distance between features at superpixelsi and j β : a scale parameter

  6. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles 3.1 Uncertainty-Based Cues 3.2 Scribble-Based Cues 3.3 Image-Level Cues 3.4 Combined Recommendation Map • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

  7. Image cues: Segment Size CodewordDistribution Recommendation Map

  8. 3.1 Uncertainty-Based Cues • 1. Node Uncertainty (NU):Fitting A1 = {GMMf,GMMb} to the labelledsuperpixel features. Using this learnt A1, for each superpixel we normalize the foreground and background likelihoods to get a 2-class distribution and then compute the entropy of this distribution. • 2. Edge Uncertainty (EU):To feed unlabelleddata- points to a set of classifiers and request label for the datapoint with maximal disagreement among classifier outcomes.

  9. 3.1 Uncertainty-Based Cues • 3. Graph-Cut Uncertainty(GC): Capture the confidence in the energy minimizing state returned by graph cuts. 3.2 Scribble-Based Cues • 4. Distance Transform over Scribbles (DT): Compute the distance of every pixel to the nearest scribble location.

  10. 3.2 Scribble-Based Cues • 5. Intervening Contours over Scribbles (IC): The value of this cue at each pixel is the maximum edge magnitude in the straight line to the closest scribble.

  11. 3.3 Image-Level Cues • 6. Segment Size (SS): When very few scribbles are marked, energy minimization methods typically overs-mooth and results in “whitewash” segmentations (entire image labelled as foreground or background). • 7. Codeword Distribution over Images (CD):Motivation being that scribbling on images containing more diversity among features would lead to better foreground /background models. To compute this cue, we cluster the features computed from all superpixels in the group to form a codebook.

  12. 3.4 Combined Recommendation Map • Learninga mapping F :φi →ϵi, • φi is the 7-dimensionalfeaturevector for superpixel i • ϵi is the error indicator vector • which is 1 if the predicted segmentation at node ϵi is incorrect, and 0 otherwise. • We chose logistic regression as the form of this mapping.

  13. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

  14. Publicly available CMU-Cornell iCoseg: 38 groups 643 images ~17 im/gp Sport

  15. The CMU-Cornell iCoseg Dataset • Dataset Annotation: The ground-truth annotations for the dataset were manually generated by a single annotator using a labelling tool.

  16. Dataset Statistics • Size • The histogram of the number of images in groups

  17. Dataset Statistics • Appearance

  18. Dataset Statistics • Scale

  19. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments 5.1 Machine Experiments 5.2 User Study • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

  20. 5.1 Machine Experiments

  21. 5.2 User Study

  22. Outline • 1 Introduction • 2 iCoseg: Energy Minimization • 3 iCoseg: Guiding User Scribbles • 4 The CMU-Cornell iCosegDataset • 5 Experiments • 6 Interactive Co-segmentation for Object-of- Interest 3D Modeling

  23. Interactive 3D modelling

  24. 3D Modeling

  25. Conclusions • iCosegthat co-segments all images in the group using an energy minimization framework, and an automatic recommendation system that intelligently recommends a region among all images in the group where the user should scribble next. Achieve good quality segmentations with significantly lower time and effortthan exhaustively examining all cutouts.

More Related