Graph Cut & Energy Minimization

1. Graph Cut &Energy Minimization Yevgeny Doctor IP Seminar 2008, IDC

2. Outline • Motivation • Graph Cuts • Energy Minimization in Computer Vision • Solving Energy Minimization with Graph Cuts • Results Yevgeny Doctor, March 2008, IDC

3. s t Graph Cut • G(V,E) is a finite directed graph and every edge (u,v) has a capacity c(u,v) (a non-negative real number). • Assume two vertices, the source s and the sink t, have been distinguished. • A cut is a split of the nodes into two sets S and T, such that s is in S and t is in T. Yevgeny Doctor, March 2008, IDC

4. Min-Cut • The capacity of a cut (S,T ) is defined as • Min-Cut – finding the cut with the minimal capacity Yevgeny Doctor, March 2008, IDC

5. Max Flow • Find the maximum flow from s to t Yevgeny Doctor, March 2008, IDC

6. Min-Cut Max-Flow • Min-Cut and Max-Flow are equivalent problems • Fast polynomial algorithms for max flow exist • For example: • Ford-Fulkerson algorithm – O(E*maxflow) • Edmonds-Karp algorithm – O(VE2) Yevgeny Doctor, March 2008, IDC

7. Energy Minimization Problems • In general terms, given some problem, we: • Formulate the known constraints • Build an “energy function” (aka “cost function”) • Look for a solution that minimizes it • If we have no further knowledge: • The problem can be NP-Hard(requires exponential solution time) • Use slow, generic approximation algorithms for optimization problems (such as simulated annealing) Yevgeny Doctor, March 2008, IDC

8. EM in Computer Vision • Consider a broad class of problems called Pixel Labeling • Given some images we want to “say something about the pixels” • For each pixel p, give it a labelfp from a finite set of labels L, such that we minimize some energy function. • Many applications • Image Segmentation • Image Restoration • Stereo and Motion • Medical Imaging • Multicamera Scene Reconstruction Yevgeny Doctor, March 2008, IDC

9. Pixel Labeling Example "Graph Cut Matching In Computer Vision", Toby Collins, February 2004 http://homepages.inf.ed.ac.uk/rbf/CVonline/ Yevgeny Doctor, March 2008, IDC

10. EM in Computer Vision • Consider a specific family of Energy Functions • Powerful enough to formulate many useful problems • Can be reduced to solving a graph min-cut problem • Problems defined with these functions: • Can be solved quickly (using max-flow algorithms) • In many cases – optimal solution or within a known factor of the optimum Yevgeny Doctor, March 2008, IDC

11. Energy Function Definition Edata(f) Esmooth(f) • Input: set of pixels P, set of labels L, is a neighbourhood system on pixels. • Goal: find a labeling that minimizes . • . is a function derived from the observed data that measures the cost of assigning label to pixel p. • . measures the cost of assigning the labels to adjacent pixels p, q. Used to impose spatial smoothness. Yevgeny Doctor, March 2008, IDC

12. Energy Function - Edata Component • The Edata(f) component: • Look at each pixel independently • Given it’s current value, what would it cost to label it with each of the labels? • Examples: • Cost based on a-priori known pixel intensity or color distribution • is the observed intensity of pixel p • What if we used only this component in E(f)? • Label each pixel independently with the most likely (cheap) label Yevgeny Doctor, March 2008, IDC

13. Optimizing Edata Only – Illustration • What would be the problem? • For example (object segmentation): Images from “Nearest Neighbor” lecture – here for illustration purposes only • We need to add a “smoothness” cost Yevgeny Doctor, March 2008, IDC

14. Energy Function - Esmooth Component • Look at all pairs of neighbor pixels • Penalize adjacent pixels with different labels • What smoothness cost function to use? Noised diamond image Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih Yevgeny Doctor, March 2008, IDC

15. Smoothness Cost Functions • Potts Interaction Penalty: • T – indicator function, K – constant • The solution will be piecewise constant, with discontinuities at the boundaries Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih

16. Smoothness Cost Functions • L2 distance: • What is the problem? • High penalties at object boundaries • We want smooth objects, but allow different labels at object boundary – a discontinuity-preserving function. Yevgeny Doctor, March 2008, IDC

17. Smoothness Cost Functions • Truncated L2 distance: • K – constant • The solution will be piecewise smooth, with discontinuities at the boundaries Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih

18. Smoothness Cost Functions • Normalize for neighbor distance, image contrast: • This function penalizes a lot for discontinuities between pixels of similar intensities when |Ip − Iq| <s. • However, if pixels are very different, |Ip − Iq| >s, then the penalty is small. Yevgeny Doctor, March 2008, IDC

19. Solving Energy Minimization with Graph Cuts • Many classes of Energy Minimization problems in Computer Vision can be reduced to Graph Cuts • Solve multiple-labels problems with binary decisions Yevgeny Doctor, March 2008, IDC

20. Approximate Energy Minimization • “Fast Approximate Energy Minimization via Graph Cuts.” Yuri Boykov, Olga Veksler, Ramin Zabih, 1999 • For two classes of interaction potentials V (Esmooth): • V is semi-metric on a label space L if for every : • V is metric on L if in addition, triangle inequality holds: • For example, truncated L2 distance and Potts Interaction Penalty are both metric. Yevgeny Doctor, March 2008, IDC

21. Solution for Semi-metric Class • Swap-Move algorithm: • 1. Start with an arbitrary labeling f • 2. Set success := 0 • 3. For each pair of labels • 3.1. Find f* = argmin E(f') among f' within one a-b swap of f • 3.2. If E(f*) < E(f), set f := f* and success := 1 • 4. If success = 1 goto 2 • 5. Return f • a-b swap: • In the new labeling f’, some pixels that were labeled a in f are now labeled b, and vice versa. Yevgeny Doctor, March 2008, IDC

22. Solve a-b swap step with Graph Cut • Graph: Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

23. Solve a-b swap step with Graph Cut • Cut and Labeling: • Weights: Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

24. Solution for Metric Class • Expansion Move algorithm: • 1. Start with an arbitrary labeling f • 2. Set success := 0 • 3. For each label • 3.1. Find f* = argmin E(f') among f' within one a expansion of f • 3.2. If E(f*) < E(f), set f := f* and success := 1 • 4. If success = 1 goto 2 • 5. Return f • a expansion: • In the new labeling some pixels change their label to a. Yevgeny Doctor, March 2008, IDC

25. Expansion Move Algorithm • Example: • Guaranteed approximation ratio by the algorithm: • Produces a labeling f such that , where f* is the global minimum and . "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

26. Solve a expansion step with Graph Cut • Graph: Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

27. Solve a expansion step with Graph Cut • Cut and Labeling: • Weights: Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

28. Experimental Results • Three energy functions: • Truncated quadratic – semi-metric (swap-move): • Potts model – metric (a-expansion): • Truncated L2 distance – metric (a-expansion): • Compared with: • Best simulated annealing algorithm Yevgeny Doctor, March 2008, IDC

29. Convergence Rate Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

30. Image Restoration Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

31. Motion and Stereo Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

32. What Energy Functions Can Be Minimized via Graph Cuts? • A precise characterization of the class of functions in F3 that can be minimized using graph cuts • A general purpose graph construction for minimizing any function in this class • A necessary condition for any function of binary variables to be minimized via graph cuts. "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

33. What Energy Functions Can Be Minimized via Graph Cuts? • Define general Energy Function classes: • Given • F2: • F3: • F2 is contained in F3 "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

34. What Energy Functions Can Be Minimized via Graph Cuts? • Regular F2 functions: "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

35. What Energy Functions Can Be Minimized via Graph Cuts? • Graph construction for F2: "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

36. What Energy Functions Can Be Minimized via Graph Cuts? • Regular F3 functions: • A function E of more than two variables is called regular if all projections of E of two variables are regular • And Generally: • Let E be a function of binary variables. If E is not regular, then E is not graph-representable. Yevgeny Doctor, March 2008, IDC

37. GrabCut "Interactive Foreground Extraction using Iterated Graph Cuts" Carsten Rother, Vladimir Kolmogorov, Andrew Blake, 2004 Microsoft Research Cambridge, UK Yevgeny Doctor, March 2008, IDC

38. GrabCut Yevgeny Doctor, March 2008, IDC

39. GrabCut • Transparency: Yevgeny Doctor, March 2008, IDC

40. GrabCut Yevgeny Doctor, March 2008, IDC

41. Video Cutout "Video Object Cut and Paste" Yin Li, Jian Sun, Heung-Yeung Shum, 2005 Microsoft Research Asia Figure 1: Given a video (a) containing an opaque video object on a complicated background, our system can generate clear alpha mattes and foreground colors (b) for the video object, which can be pasted onto another background. Yevgeny Doctor, March 2008, IDC

42. Video Cutout • 3D segmentation of spatio-temporal video volume • Partition into foreground and background, while preserving temporal coherence • Local refinement of initial segmentation • Provide brush tools for the user to control the object boundary precisely wherever needed • Apply coherent matting to extract the alpha mattes and foreground colors of the object based on the accurate binary segmentation result Yevgeny Doctor, March 2008, IDC

43. Video Cutout Yevgeny Doctor, March 2008, IDC

44. Video Cutout Yevgeny Doctor, March 2008, IDC

45. Video Cutout Yevgeny Doctor, March 2008, IDC

46. Video Cutout Figure 7: Clip #2, frame 27. (a) 3D graph cut result is shown by the overlaid dashed line. The flag is a rapidly deforming object, but 3D graph cut can apture the shape very well. (b) Dashed lines indicate the boundaries after both the local refinement and overriding operations. The white pixels record the actions of foreground brushing and the black pixels for background brushing. (c) coherence matting result pasted on a blue screen. Yevgeny Doctor, March 2008, IDC