Interactive Graph Cuts for Segmentation in N-D Images

1. Interactive Graph Cuts for Segmentation in N-D Images Yuri Boykov, Marie-Piere Jolly Mohit Gupta 02/15/2006 Advanced Perception

2. Image Segmentation Images: Label Me Database

3. Focus of this work: Foreground vs. Background • **The framework presented can be easily extended to multi-label segmentation though • **Multi-label segmentation solved by iteratively solving many 2-label • sub-problems (Boykov, Veksler, Zabih) Images: Yin Li et al (Lazy Snapping)

4. Focus of this work: Foreground vs. Background • **The framework presented can be easily extended to multi-label segmentation though • **Multi-label segmentation solved by iteratively solving many 2-label • sub-problems (Boykov, Veksler, Zabih) Supervised Method Images: Yin Li et al (Lazy Snapping)

5. Fully Automatic Methods: EigenVectors Based approaches (Normalized Cuts, Perona and Freeman Algorithm etc.) Supervised Methods Boundary Based approaches: use of an ‘evolving curve’ (snakes, intelligent scissor, image snapping) Region Based approaches: hints about foreground vs. background (intelligent paint, GrabCut) Taxonomy + Previous Work

6. Fully Automatic Methods: EigenVectors Based approaches (Normalized Cuts, Perona and Freeman Algorithm etc.) Supervised Methods Boundary Based approaches: use of an ‘evolving curve’ (snakes, intelligent scissor, image snapping) Region Based approaches: hints about foreground vs. background (intelligent paint, GrabCut) Edge Weights depend on the ‘boundary-ness’ High probability of boundary  Low weight Taxonomy + Previous Work

7. Fully Automatic Methods: EigenVectors Based approaches (Normalized Cuts, Perona and Freeman Algorithm etc.) Supervised Methods Boundary Based approaches: use of an ‘evolving curve’ (snakes, intelligent scissor, image snapping) Region Based approaches: hints about foreground vs. background (intelligent paint, GrabCut) Boundary = Shortest Path between graph vertices Taxonomy + Previous Work

8. Fully Automatic Methods: EigenVectors Based approaches (Normalized Cuts, Perona and Freeman Algorithm etc.) Supervised Methods Boundary Based approaches: use of an ‘evolving curve’ (snakes, intelligent scissor, image snapping) Region Based approaches: hints about foreground vs. background (intelligent paint, GrabCut) Taxonomy + Previous Work

9. Case for a Supervised, Region Based Method • Fully Automatic • Never perfect – Hard to get crisp boundaries for inherently ambiguous, low contrast images • Boundary Based Supervised • Complex Object: User’s nightmare "Whenever something can be done in two ways, someone will be confused. Whenever something is a matter of taste, discussions can drag on forever." -- Bjarne Stroustrup

10. Supervised Region based segmentation technique What user does?: Provide clues as to the ‘desired’ segmentation Foreground and Background ‘seed pixels’ Energy of the segmentation: Lower the energy, better the segmentation Interactive Graph Cuts: Overview

11. Problem Formulation • Input • Set of pixels P, • Neighborhood system N {p,q} • Hard Constraints: ‘clues’ for segmentation • Objective function: Soft Constraint (Energy) • Output • Assignment vector A = (A1, … , Ap , … , A|P|), where Aie {0,1} • A defines a segmentation of the image

12. Objective Function:Energy of the Segmentation TO DO: Minimize Energy E(A) while satisfying the hard constraints • Regional Term R(A)  Cost for assigning labels to individual pixels • Boundary Term B(A)  Cost for making the boundary pass between • two given pixels

13. Graph Cuts and Image Segmentation A node for every pixel Two terminal nodes n-links and t-links Edge weights: n-links Image with seeds t-link Background Terminal t t-link Object Terminal Corresponding Graph s

14. Graph Cut Segmentation A (A1, … , Ap , … , A|P|) a cut n-links n-links Image with seeds Segmentation Results t-link t-link Background Terminal t t t-link t-link Object Terminal Graph Cut Corresponding Graph s s Min-Cut Energy Minimization

15. Graph Cut Segmentation A (A1, … , Ap , … , A|P|) **|C| = E(A)** Min |C| = min E(A) a cut n-links n-links Image with seeds Segmentation Results t-link t-link Background Terminal t t t-link t-link Object Terminal Graph Cut Corresponding Graph s s Min-Cut Energy Minimization

16. Graph Cut Segmentation A (A1, … , Ap , … , A|P|) **|C| = E(A)** Min |C| = min E(A) a cut n-links n-links Image with seeds Segmentation Results t-link t-link Background Terminal t t t-link t-link Object Terminal Graph Cut Corresponding Graph s s Min-Cut Energy Minimization Min-cut ‘Optimal’ Segmentation!

17. Results • Low values of l • Boundary term dictates • Region “Shrinking” • High values of l • Region term dictates • Neighborhood info ignored

18. We don’t want toy results… • Start with a few ‘obj’ and ‘bckg’ seeds • Refinement in trouble places Images: Yin Li et al (Lazy Snapping)

19. Interactivity New ‘obj’/’bckg’ pixel added/deleted Only two links change: Real time re-computation Possible  Interactive!

20. More Results Images: Yin Li et al (Lazy Snapping), Boykov and Jolly

21. Performance • Efficient polynomial time algorithms available for min-cut • Segmentation Problem  Sparse graphs • Computations in the “blink of an eye” • Efficiency measured in terms of amount of human effort • Bell Example takes about a minute

22. Markov Random Fields (MRF’s) • S = discrete set of sites S = {1, …, m} • Ld = discrete set of labels, eg. {1, … L}. • Neighborhood system N • A labeling assigns a label to every site, f = {f1, … fm}. fi is the label of site i. • Neighborhood  conditional independence • F is an MRF on S w.r.t. N iff: • P(f) > 0 • P(fi | fS-{i}) = P(fi | fNi)

23. Lazy Snapping (SIGGRAPH’2004) Yin Li, Jian Sun, Chi-Keung Tang, Heung-Yeung Shum(MSRA) • ‘Refine’ the coarse results returned by Graph-Cut Based methods (Boykov and Jolly) • Extremely Fast • Commercial System  User Friendly

24. Modus Operandi: How does it work? • Supervised Region Based Method • Three Step Process • Pre-segmentation ( super pixels ) • Object Marking Step ( a la Boykov and Jolly) • Local Refinement Step

25. Pre-segmentation • Exploit Spatial Coherency  Replace Pixels by Super-Pixels • ‘Aggressive’ unsupervised segmentation  preserves color coherency within regions

26. Pre-segmentation

27. Boundary Editing • Object Boundary from previous step represented as polygon • Polygon can be edited for local refinement • Local refinement  New constraints • Another optimization for better fit

28. Boundary Editing • Dij term penalizes a node pair far away from the polygon  brings boundary closer to the polygon b = 1  boundary snaps to polygon **Important: The optimized boundary snaps to the object boundary even though the polygon vertices may not be on it.

29. citius altius fortius… • Faster, easier, accurat’er’

30. The pretty SIGGRAPH video

31. Multi-way graph cuts(Boykov, Veksler, Zabih) Images: Label Me Database

32. Multi-way graph cuts Slides: Yuri Boykov

33. Multi-way graph cuts BAD NEWS: • NP-hard problem (3 or more labels) • two labels can be solved in polynomial time via s-t cuts GOOD NEWS: • a-expansion approximation algorithm • guaranteed approximation quality (2-approx) Slides: Yuri Boykov

34. other labels a a-expansion move Basic idea: break multi-way cut computation into a sequence of binary s-t cuts Slides: Yuri Boykov

35. other labels a a-expansion move Basic idea: break multi-way cut computation into a sequence of binary s-t cuts **Iteratively, each label competes with other labels for space in the image

36. a-expansion algorithm • Start with any initial solution • For each label “a” in any (e.g. random) order • Compute optimal a-expansion move (s-t graph cuts) • Decline the move if there is no energy decrease • Stop when no expansion move would decrease energy Slides: Yuri Boykov

37. a-expansion algorithm • Start with any initial solution • For each label “a” in any (e.g. random) order • Compute optimal a-expansion move (s-t graph cuts) • Decline the move if there is no energy decrease • Stop when no expansion move would decrease energy A 2-approx algorithm for an NP-complete problem! Converges in a few iterations!

38. Video Segmentation Video Object Cut and Paste (SIGGRAPH’2004) Yin Li, Jian Sun, Heung-Yeung Shum(MSRA)

39. Video Segmentation: 3D graph on volume of frames Edges for ‘temporal coherence’ in addition to ‘spatial coherence’ ‘obj’ + ‘bckg’ seeds on a few key-frames Climbing the dimension ladder2D  3D Image: Yin Li et al

40. Additional term for temporal neighbors What’s new? Temporal Coherence Our old friend…

41. Local Refinement: Problem • ‘obj’ / ‘bckg’ color models built globally from key frames • Might get confused between the two !

42. Local Refinement: Solution • Mark windows around problem areas in key frames • Windows propagate through middle frames using a feature tracking algorithm • Apply 2D pixel level graph cut segmentation • Important: Seeds generated automatically

43. Video Segmentation: Results Images: Boykov and Jolly

44. Another (obviously pretty) SIGGRAPH video

45. Conclusion • Merits • Globally Optimum Segmentation, when cost function is clearly defined (normalized cuts only give an approximate solution) • Very Fast: Interactive Rates • Natural Extension to N-Dimensional images • Easily Editable: More user friendly • Possible Improvement • Automatic Seed Selection: From Coarse (LabelMe) kind of segmentation