Fast Interactive Image Segmentation by Discriminative Clustering

1. Fast Interactive Image Segmentation by Discriminative Clustering Dingding Liu * Kari Pulli † Linda Shapiro * YingenXiong † † Nokia Research Center, Palo Alto, CA 94304, USA *Dept. Elect. Eng., University of Washington, WA 98095, USA

2. Research Aim • Cut out an object from its background fast • Computation time – so can quickly iterate • With as few strokes as possible

3. Overview • Introduction • Motivation • Related work • Algorithm • Pre-segmentation by the Mean-Shift algorithm • Merge regions by discriminative clustering • Local neighborhood region classification and pruning • Experiments and Results • Conclusions and Future Work

4. Introduction • Motivation: Image editing on mobile devices • Convenience • Anytime, anywhere • Challenges • Limited computational resources • Smaller screens and imprecise input

5. Related Work -Interactive Image Segmentation • Lazy Snapping • Li et al., ACM Transactions on Graphics 2004

6. Related Work -Interactive Image Segmentation • Interactive Image Segmentation by Maximal Similarity Based Region Merging • Ning et al., Pattern Recognition 2010 Insufficient user inputs Sufficient user inputs

7. Algorithm: Summary Pre-segmentation by the Mean-Shift algorithm Merge regions by discriminative clustering Local neighborhood region classification and pruning

8. Background: Mean-Shift Segmentation http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html http://robots.stanford.edu/cs223b04/CS%2520223-B%2520L11%2520Segmentation.ppt

9. The Basic Mean-Shift Algorithm • Choose a search window size • Choose the initial location of the search window • Compute the mean (centroid of the data) within the search window • Center the search window at that mean location • Repeat 3 and 4 until convergence The mean shift algorithm seeks the“mode”or point of highest density of a data distribution

10. Mean-Shift Segmentation • Convert the image into tokens (via color, gradients, texture measures, etc.) • Choose initial search window locations uniformly in the data • Compute the mean shift window location for each initial position • Merge windows that end up on the same “peak” or mode • Repeat 3 and 4 until convergence

11. Mean-Shift Segmentation Results

12. Algorithm: Pre-segmentation using Mean-Shift • Three reasons for choosing the Mean-Shift algorithm: • 1. It preserves the boundaries better than other methods • 2. Its speed has been improved significantly in recent years Pre-segmentation can be done either before or after the user input • 3. Fewer parameters to tune

13. Algorithm: Merge non-ambiguous regions df > dthresh + db, background df + dthresh < db, foreground Otherwise, ambiguous regions Only consider color, not location • Create two kd-trees in CIELab color space • One for the marked foreground, another for the background regions • For each unmarked region, find the color difference to • the most similar marked background dband foreground region df • Choice of dthresh : • Min difference of mean colors between the marked foreground and background • that is higher than a minimum allowed distance (we chose 2)

14. Algorithm: Assign ambiguous regions • Now use also location information • Each of the remaining ambiguous regions is assigned • the label of the neighboring region that has the most similar mean color • If the most similar neighboring region is also an unmarked region • merge them to a new unmarked region, repeat the process • If there is a tie in the mean color for assignment to foreground and background • the label of the region that has the most similar color variance is used

15. Algorithm: Prune / flip isolated regions Find isolated foreground or background regions (use connected components) • Regions are changed to the opposite label when all of the following hold: • The region is not marked by the user (b) The region is not the biggest region with that label (c) The region is smaller than its surrounding regions

16. Results: Segmentation time – in numbers

17. Results: Segmentation time – as a graph

18. Why are we faster? • Two main reasons • No iterative steps in the first stage • and not too many in the second or third • do the easy choices quickly • fast nearest-neighbor lookups with kd-trees • graph-cut on many regions is slow, MSRM iterates unnecessarily much • Merging the region descriptor is fast • only mean and standard deviation of colors • MSRM has complicated 4K bin color histograms to merge

19. Results: Segmentation time on phone

20. Results: The best segmentation quality (a) Input image (c) Maximal Similar Region Merging (d) Proposed method (b) Graph-cut over regions

21. Results: The best segmentation quality (b) Graph-cut over regions (c) Maximal Similar Region Merging (d) Proposed method (a) Input image

22. Results: The best segmentation quality (b) Graph-cut over regions (c) Maximal Similar Region Merging (d) Proposed method (a) Input image

23. Results: Video Demo

24. Conclusions and Future Work • A new region-based interactive image segmentation algorithm • Significantly increases the speed of segmentation • by avoiding global optimization and long iterations • Does not compromise the segmentation quality • Uses a region mean color instead of a single pixel color • Future Work • Further decrease the users input • Combine the individual pixel information to further improve the algorithm

25. Thank you! • Questions?