Adaptation from:

1. Adaptation from: Prof. James M. Rehg, G.Tech Computer Vision : CISC 4/689

2. Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923 Computer Vision : CISC 4/689

3. Teesta suspension bridge-Darjeeling, India Computer Vision : CISC 4/689

4. Mark Twain at Pool Table", no date, UCR Museum of Photography Computer Vision : CISC 4/689

5. Woman getting eye exam during immigration procedure at Ellis Island, c. 1905 - 1920, UCR Museum of Phography Computer Vision : CISC 4/689

6. Stereo results • Data from University of Tsukuba Scene Ground truth (Seitz) Computer Vision : CISC 4/689

7. Results with window correlation Window-based matching (best window size) Ground truth (Seitz) Computer Vision : CISC 4/689

8. Results with better method • State of the art method • Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, • International Conference on Computer Vision, September 1999. Ground truth Computer Vision : CISC 4/689 (Seitz)

9. Stereo Constraint • Color constancy • The color of any world points remains constant from image to image • This assumption is true under Lambertian Model • In practice, given photometric camera calibration and typical scenes, color constancy holds well enough for most stereo algorithms. Computer Vision : CISC 4/689

10. Stereo Constraint • Epipolar geometry • The epipolar geometry is the fundamental constraint in stereo. • Rectification aligns epipolar lines with scanlines Epipolar plane Epipolar line for p Computer Vision : CISC 4/689 Epipolar line for p’

11. Stereo Constraint • Uniqueness and Continuity • Proposed by Marr&Poggio. • Each item from each image may be assigned at most one disparity value,” and the “disparity” varies smoothly almost everywhere. Computer Vision : CISC 4/689

12. Problems with Window-based matching • Disparity within the window must be constant. • Bias the results towards frontal-parallel surfaces. • Blur across depth discontinuities. • Perform poorly in textureless regions. • Erroneous results in occluded regions Computer Vision : CISC 4/689

13. Cooperative Stereo Algorithm • Based on two basic assumption by Marr and Poggio: • Uniqueness: at most a single unique match exists for each pixel. • Continuous: disparity values are generally continuous, i.e., smooth within a local neighborhood. Computer Vision : CISC 4/689

14. Disparity Space Image (DSI) • The 3D disparity space has dimensions row r column c and disparity d. Each element (r, c, d) of the disparity space projects to the pixel (r, c) in the left image and to the (r, c + d) in the right image • DSI represents the confidence or likelihood of a particular match. Computer Vision : CISC 4/689

15. Illustration of DSI (r, c) slices for different d (c, d) slice for r = 151 Computer Vision : CISC 4/689

16. Definition Match value assigned to element (r, c, d) at iteration n Initial values computed from SSD or NCC Inhibition area for element (r, c, d) Local support area for element (r, c, d) Computer Vision : CISC 4/689

17. Illustration of Inhibitory and Support Regions If d is disparity for (r,c), d -1 is for (r,c-1) And d+1 for (r,c+1), etc. Vertical inhibition is self explanatory. Diagonal basically means, r,c+1in left image cannot map to r,c+d in right image, i.e, it cannot have disparity of d-1. Similarly, r,c-1 in left image cannot map to r,c+d so it cannot have disparity of d+1. This satisfies uniqueness constraint. Computer Vision : CISC 4/689

18. Iterative Updating DSI 1 2 • Questions to ponder: • What is typical alpha? • Do we include inhibition • region in numerator of Rn? 3 4 Computer Vision : CISC 4/689

19. Explicit Detection of Occlusion Identify occlusions by examining the magnitude of the converged values in conjunction with the uniqueness constrain Computer Vision : CISC 4/689

20. Summary of Cooperative Stereo • Prepare a 3D array, (r, c, d): (r, c) for each pixel in the reference image and d for the range of disparity. • Set initial match values using a function of image intensities, such as normalized correlation or SSD. • Iteratively update match values using (4) until the match values converge. • For each pixel (r, c), find the element (r, c, d) with the maximum match value. • If the maximum match value is higher than a threshold, output the disparity d, otherwise, declare a occlusion. Computer Vision : CISC 4/689

21. Ordering Constraint • If an object a is left on an object b in the left image then object a will also appear to the left of object b in the right image Ordering constraint… …and its failure Computer Vision : CISC 4/689

22. … … Stereo Correspondences Left scanline Right scanline Match intensities sequentially between two scanlines Computer Vision : CISC 4/689

23. … … Match Match Match Left occlusion Right occlusion Stereo Correspondences Left scanline Right scanline Computer Vision : CISC 4/689

24. Left Occluded Pixels Search Over Correspondences Three cases: • Sequential – cost of match • Left occluded – cost of no match • Right occluded – cost of no match Left scanline Right scanline Right occluded Pixels Computer Vision : CISC 4/689

25. Left Occluded Pixels Standard 3-move Dynamic Programming for Stereo Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Left scanline Start Right scanline Right occluded Pixels Computer Vision : CISC 4/689 End

26. Dynamic Programming • Efficient algorithm for solving sequential decision (optimal path) problems. 1 1 1 1 … 2 2 2 2 3 3 3 3 How many paths through this trellis? Computer Vision : CISC 4/689

27. Dynamic Programming 1 1 1 States: 2 2 2 3 3 3 Suppose cost can be decomposed into stages: Computer Vision : CISC 4/689

28. Dynamic Programming 1 1 1 2 2 2 3 3 3 Principle of Optimality for an n-stage assignment problem Computer Vision : CISC 4/689

29. Dynamic Programming 1 1 1 2 2 2 3 3 3 Computer Vision : CISC 4/689

30. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

31. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

32. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

33. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

34. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

35. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

36. Occluded Pixels Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path. For each (i,j), look for what is Optimal to get to: would it be From (i-1,j-1), or (i,j-1), or (i-1,j-1)? Assign large values for left occlusion and right occlusion. Left scanline Dis-occluded Pixels Right scanline Terminal Computer Vision : CISC 4/689

37. Stereo Matching with Dynamic Programming Pseudo-code describing how to calculate the optimal match Computer Vision : CISC 4/689

38. Stereo Matching with Dynamic Programming Pseudo-code describing how to reconstruct the optimal path Computer Vision : CISC 4/689

39. Results Local errors may be propagated along a scan-line and no inter scan-line consistency is enforced. Computer Vision : CISC 4/689

40. Assumption Behind Segmentation-based Stereo • Depth discontinuity tend to correlate well with color edges • Disparity variation within a segment is small • Approximating the scene with piece-wise planar surfaces Computer Vision : CISC 4/689

41. Segmentation-based stereo • Plane equation is fitted in each segment based on initial disparity estimation obtained SSD or Correlation • Global matching criteria: if a depth map is good, warping the reference image to the other view according to this depth will render an image that matches the real view • Optimization by iterative neighborhood depth hypothesizing Computer Vision : CISC 4/689

42. Result Computer Vision : CISC 4/689

43. Another Result Computer Vision : CISC 4/689