Stereo & Iterative Graph-Cuts

1. Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University

2. The stereo problem

3. The stereo problem • Both images are very similar (like images that you see with your two eyes) • • Most of the pixels in the left image are present in the right image (except for few occlusions)

4. The stereo problem After rectification: all correspondences are along the same horizontal scan lines (pixels in one image simply shift horizontally in the other image)

5. The stereo problem • The horizontal shifts between the images are sometimes called: “disparities” • The Disparities relates to depth: Closer objects have larger disparities.

6. The stereo problem: compute the disparity map between two images

7. Traditional Approaches • Matching rigid windows around each pixel • Each window is matched independently Modern approaches • Finding coherent correspondences for all pixels - “Graph cuts” - “Belief Propagation”

8. Window-Based Approach • Compute a cost for each location • Location with the lowest cost wins

9. General Problem : Ambiguity Left Right scanline

10. Window-Based Approach Small Window Large Window blurred boundaries noisy in low texture areas

11. Results with best window size(still not good enough) Window-based matching (best window size) Ground truth

12. Graph Cuts Graph cuts Ground truth

13. Max flow problem: Each edge is a “pipe” Find the largest flow F of “water” that can be sent from the “source” to the “sink” along the pipes Edge weights give the pipe’s capacity a flow F “source” “sink” T S A graph with two terminals Maximum flow problem

14. Min cut problem: Find the cheapest way to cut the edges so that the “source” is completely separated from the “sink” Edge weights now represent cutting “costs” a cut C “source” “sink” T S A graph with two terminals Minimum cut problem

15. Max Flow = Min Cut: Maximum flow saturates the edges along the minimum cut. Ford and Fulkerson, 1962 Problem reduction! Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution “source” “sink” T S A graph with two terminals Max flow/Min cut theorem

16. Min-Cut: Important Rule No subset of the cut can also be a cut This is not a minimal cut

17. Energy Minimization Using Iterative Graph cuts Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler and Ramin ZabihPami 2001

18. To do better we need a better model of images • We can make reasonable assumptions about the surfaces in the world • Usually assume that the surfaces are smooth • Can pose the problem of finding the corresponding points as an energy (or cost) minimization: f-assignment how well the pixels match up for different disparities neighboring pixels have similar disparities

19. To do better we need a better model of images • We can make reasonable assumptions about the surfaces in the world • Usually assume that the surfaces are smooth • Can pose the problem of finding the corresponding points as an energy (or cost) minimization: f-assignment p,q-pixels Data term is calculated for each pixels Smoothness is calculated on neighbor pixels

20. Example for Smoothness terms • Quadratic • L1 • Truncated L1 • Potts model

21. Constructing a Graph to Solve the Stereo Problem

22. Constructing a Graph to Solve the Stereo Problem

23. Constructing a Graph to Solve the Stereo Problem The labels of each pixel are the possible disparity values

24. Constructing a Graph to Solve the Stereo Problem The labels of each pixel are the possible disparity values

25. Smoothness term Relation between the Energy and the Graph labeling problem {fp=10} Data term 10 1 {fq=2} q p

26. Smoothness term Relation between the Energy and the Graph labeling problem Dp(10) Data term 10 V(p,q)(1, 10) 1 q p

27. Iterative graph-cuts • Use an iterative scheme to find a “good” local optimum of the energy function. • In each iteration: convert the original multi-label problem to a binary one, and solve it by finding a minimal graph-cut (max-flow). • The most popular scheme is the expansion move. • -expansion: set the label of each pixel to be either or the current label.

28. Types of Moves A Single Pixel Move Problem: A lot of local minima

29. Types of Moves Expansion Move Any pixel can change its label to alpha

30. Types of Moves Expansion Move Claim (without proof): The difference between the optimal solution and the solution from the iterative expansion moves is bounded

31. Energy Minimization Algorithm • Start with arbitrary labeling f • Set success = 0 • For each label • Find • If set and success =1 • If success =1 goto (2) • Return f

32. Conditions on the Smoothness for using expansion moves: In other words: V should be a metric Note : The Quadratic smoothness is not a metric

33. For each pair of vertices such that we add a ‘dummy’ vertex (together with the respective edges as shown in the table).

34. The Relation between the cut and the Energy • Given a cut C, we define a labelling fc by: • The cost of a cut C is |C| = E(fC) (plus a constant) If the cut C separates p and If the cut C separates p and

35. The Relation between the cut and the Energy

36. The Relation between the cut and the Energy

37. Conditions on the Smoothness for using expansion moves: In other words: V should be a metric