Photomontage

1. GrabCut Interactive Image(and Stereo) Segmentation Carsten RotherVladimir Kolmogorov Andrew BlakeAntonio CriminisiGeoffrey Cross[based on Siggraph 2004 and ECCV 2004]

2. GrabCut – Interactive Foreground Extraction1 Photomontage

3. GrabCut – Interactive Foreground Extraction2 Talk Outline Hard Image Segmentation: Fore- vs. Background Soft Segmentation: Alpha Matting Stereo Segmentation: Exploit Depth

4. GrabCut – Interactive Foreground Extraction3 Problem Fast & Accurate ?

5. GrabCut – Interactive Foreground Extraction4 What GrabCut does Magic Wand(198?) Intelligent ScissorsMortensen and Barrett (1995) GrabCut User Input Result Regions Regions & Boundary Boundary

6. GrabCut – Interactive Foreground Extraction5 Framework Input: Image Output: Segmentation Parameters:Colour ,Coherence Energy: Optimization:

7. GrabCut – Interactive Foreground Extraction6 Energy – Probabilistic View • Gibbs Distribution of the MRF Maximum a posteriori estimator (MAP): - log same as

8. Foreground (source) Min Cut Background(sink) Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time GrabCut – Interactive Foreground Extraction7 Graph Cuts - Boykov and Jolly (2001) Image

9. GrabCut – Interactive Foreground Extraction8 Iterated Graph Cut ? User Initialisation Graph cuts to infer the segmentation K-means for learning colour distributions

10. GrabCut – Interactive Foreground Extraction9 Iterated Graph Cuts Guaranteed toconverge 1 3 4 2 Result Energy after each Iteration

11. GrabCut – Interactive Foreground Extraction10 Colour Model Gaussian Mixture Model (typically 5-8 components) R R Iterated graph cut Foreground &Background Foreground G Background G Background

12. Coherence Model An object is a coherent set of pixels: 25 Error (%) over training set: How do we choose ? 25

13. GrabCut – Interactive Foreground Extraction12 Parameter Learning (Blake 2004) Gaussian MRF: approximation = Pseudo-Likelihood: Linear regression gives in closed-form

14. GrabCut – Interactive Foreground Extraction13 Parameter Learning - Problems A Gaussian MRF is not a realistic texture model syntheticGMRF Gaussian? Real Image Gaussian!

15. GrabCut – Interactive Foreground Extraction14 Moderately simple examples … GrabCut completes automatically

16. GrabCut – Interactive Foreground Extraction15 Difficult Examples Camouflage & Low Contrast Fine structure No telepathy Initial Rectangle InitialResult

17. GrabCut – Interactive Foreground Extraction16 Evaluation – Labelled Database Available online: http://research.microsoft.com/vision/cambridge/segmentation/

18. Error Rate: 0.72% GrabCut – Interactive Foreground Extraction17 Comparison Boykov and Jolly (2001) GrabCut User Input Result Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.32% Error Rate: 1.25% Error Rate: 0.72%

19. GrabCut – Interactive Foreground Extraction18 Comparison BimapGrabCut Error Rate: 2.13% Input Image Ground Truth Trimap Boykov and Jolly Error Rate: 1.36% Error rate - modestly increase User Interactions - considerable reduced

20. GrabCut – Interactive Foreground Extraction19 Results Parameter Learning

21. GrabCut – Interactive Foreground Extraction20 Comparison Intelligent Scissors Mortensen and Barrett (1995) LazySnappingLi et al. (2004) Graph Cuts Boykov and Jolly (2001) Magic Wand (198?) GrabCutRother et al. (2004)

22. GrabCut – Interactive Foreground Extraction21 Digital Matting • “Mixed pixels”: Combination of fore- and background • Alpha Mask: Proportion of fore- and background • Natural Matting Problem: Determine alpha,F,B from C Under-determined System: 3 Equations and 7 unknowns

23. Existing Methods GrabCut Human ? GrabCut – Interactive Foreground Extraction22 Transparency - Taxonomie 1. Simple Alpha & Simple Colour 2. Difficult Alpha & Simple Colour 3. Simple Alpha & Difficult Colour 4. Difficult Alpha & Difficult Colour

24. GrabCut – Interactive Foreground Extraction23 Border Matting Hard Segmentation Automatic Trimap Soft Segmentation to

25. GrabCut – Interactive Foreground Extraction24 Comparison With no regularisation over alpha Input Knockout 2Photoshop Plug-In Bayes MattingChuang et. al. (2001) Shum et. al. (2004):Coherence matting in “Pop-up light fields”

26. GrabCut – Interactive Foreground Extraction25 Natural Image Matting Mean Colour Foreground Mean ColourBackground Solve Ruzon and Tomasi (2000):Alpha estimation in natural images

27. GrabCut – Interactive Foreground Extraction26 Border Matting Foreground Noisy alpha-profile 1 Mix Back-ground 0 Foreground Background Mix Fit a smooth alpha-profile with parameters

28. GrabCut – Interactive Foreground Extraction27 Dynamic Programming t+1 t DP Result using DP Border Matting Regularisation Noisy alpha-profile

29. GrabCut – Interactive Foreground Extraction28 GrabCut BorderMatting -Colour • Compute MAP of p(F|C,alpha) (marginalize over B) • To avoid colour bleeding use colour stealing (“exemplar based inpainting” – Patches do not work) Grabcut Border Matting [Chuang et al. ‘01]

30. GrabCut – Interactive Foreground Extraction29 Results

31. GrabCut – Interactive Foreground Extraction30 Stereo Video + Segmentation Right Sequence Left Sequence Criminisi et. al. (2003):4-Plane DP to handle occlusions properly Disparity Sequence

32. GrabCut – Interactive Foreground Extraction31 Occusion, left and right

33. GrabCut – Interactive Foreground Extraction32 Background Substitution Criminisi et. al. (2004): Remove boundary artefacts (SPS algorithm)

34. GrabCut – Interactive Foreground Extraction33 Object Insertion

35. GrabCut – Interactive Foreground Extraction34 Focus on Foreground

36. GrabCut – Interactive Foreground Extraction35 Conclusions & Future Work GrabCut – powerful interactive extraction tool Iterated Graph Cut based on colour and contrast Regularized alpha matting by Dynamic Programming Stereo and Segmentation give supportive information How to solve the difficult hair problem ? [Argawall et.al.2004]