1 / 88

Fundamentals of Stereo Vision

What Happened Last Time?. Human 3D perception (3D cinema)Computational stereoIntuitive explanation of what is meant by disparityStereo matching problemVarious applications of stereo. What Is Going to Happen Today?. Stereo from a technical point of view:Stereo pipelineEpipolar geometryEpipola

keala
Download Presentation

Fundamentals of Stereo Vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


    1. Fundamentals of Stereo Vision Michael Bleyer LVA Stereo Vision

    2. What Happened Last Time? Human 3D perception (3D cinema) Computational stereo Intuitive explanation of what is meant by disparity Stereo matching problem Various applications of stereo

    3. What Is Going to Happen Today? Stereo from a technical point of view: Stereo pipeline Epipolar geometry Epipolar rectification Depth via triangulation Challenges in stereo matching Commonly used assumptions Middlebury stereo benchmark

    4. Michael Bleyer LVA Stereo Vision

    5. Stereo Pipeline

    6. Stereo Pipeline

    7. Stereo Pipeline

    8. Stereo Pipeline

    9. Pinhole Camera Simplest model for describing the projection of a 3D scene onto a 2D image. Model is commonly used in computer vision.

    10. Image Formation Process Let us assume we have a pinhole camera. The pinhole camera is characterized by its focal point Cl and its image plane L.

    11. Image Formation Process We also have a second pinhole camera <Cr,R>. We assume that the camera system is fully calibrated, i.e. the 3D positions of <Cl, L> and <Cr,R> are known.

    12. Image Formation Process We have a 3D point P.

    13. Image Formation Process We compute the 2D projection pl of P onto the image plane of the left camera L by intersecting the ray from Cl to P with the plane L. This is what is happening when you take a 2D image of a 3D scene with your camera (image formation process).

    14. Image Formation Process We compute the 2D projection pl of P onto the image plane of the left camera L by intersecting the ray from Cl to P with the plane L. This is what is happening when you take a 2D image of a 3D scene with your camera (image formation process).

    15. 3D Reconstruction Task: We have a 2D point pl and want to compute its 3D position P.

    16. 3D Reconstruction P has to lie on the ray of Cl to pl. Problem: It can lie anywhere on this ray.

    17. 3D Reconstruction Let us assume we also know the 2D projection pr of P onto the right image plane R.

    18. 3D Reconstruction P can now be reconstructed by intersecting the rays Clpl and Crpr.

    19. 3D Reconstruction P can now be reconstructed by intersecting the rays Clpl and Crpr.

    20. 3D Reconstruction P can now be reconstructed by intersecting the rays Clpl and Crpr.

    21. Epipolar Geometry We have stated that P has to lie on the ray Clpl.

    22. Epipolar Geometry If we project each candidate 3D point onto the right image plane, we see that the all lie on a line in R.

    23. Epipolar Geometry If we project each candidate 3D point onto the right image plane, we see that the all lie on a line in R.

    24. Epipolar Geometry If we project each candidate 3D point onto the right image plane, we see that the all lie on a line in R.

    25. Epipolar Geometry If we project each candidate 3D point onto the right image plane, we see that the all lie on a line in R.

    26. Epipolar Geometry This line is called epipolar line of pl. This epipolar line is the projection of the ray Clpl onto the right image plane R. The pixel pr is forced to lie on pls epipolar line.

    27. Epipolar Geometry This line is called epipolar line of pl. This epipolar line is the projection of the ray Clpl onto the right image plane R. The pixel pr is forced to lie on pls epipolar line.

    28. Epipolar Rectification Specifically interesting case: Image plane L and R lie in a common plane. X-axes are parallel to the baseline Epipolar lines coincide with horizontal scanlines => corresponding pixels have the same y-coordinate

    29. Epipolar Rectification Specifically interesting case: Image plane L and R lie in a common plane. X-axes are parallel to the baseline Epipolar lines coincide with horizontal scanlines => corresponding pixels have the same y-coordinate

    30. Epipolar Rectification Specifically interesting case: Image plane L and R lie in a common plane. X-axes are parallel to the baseline Epipolar lines coincide with horizontal scanlines => corresponding pixels have the same y-coordinate

    31. Epipolar Rectification Specifically interesting case: Image plane L and R lie in a common plane. X-axes are parallel to the baseline Epipolar lines coincide with horizontal scanlines => corresponding pixels have the same y-coordinate

    32. Epipolar Rectification Specifically interesting case: Image plane L and R lie in a common plane. X-axes are parallel to the baseline Epipolar lines coincide with horizontal scanlines => corresponding pixels have the same y-coordinate

    33. Epipolar Rectification This special case can be achieved by reprojecting left and right images onto virtual cameras. This process is known as epipolar rectification. Throughout the rest of the lecture we assume that images have been rectified.

    34. Epipolar Constraint Concluding Remarks Epipolar constraint should always be used, because: 1D search is computationally faster than 2D search. Reduced search range lowers chance of finding a wrong match (Quality of depth maps). More or less the only constraint that will always be valid in stereo matching (unless there are calibration errors).

    35. Stereo Pipeline

    36. Depth via Triangulation

    37. Depth via Triangulation

    38. Depth via Triangulation

    39. Depth via Triangulation

    40. Depth via Triangulation

    41. Depth via Triangulation

    42. Depth via Triangulation

    43. Depth via Triangulation

    44. Depth via Triangulation From similar triangles: Write X in explicit form: Combine both equations: Write Z in explicit form:

    45. Depth via Triangulation From similar triangles: Write X in explicit form: Combine both equations: Write Z in explicit form:

    46. Stereo Pipeline

    47. Michael Bleyer LVA Stereo Vision

    48. Stereo Matching

    49. Why is Stereo Matching Challenging? (1) Color inconsistencies: When solving the stereo matching problem, we typically assume that corresponding pixels have the same intensity/color (= Photo consistency assumption) That does not need to be true due to: Image noise Different illumination conditions in left and right images Different sensor characteristics of the two cameras. Specular reflections (mirroring) Sampling artefacts Matting artefacts

    50. Why is Stereo Matching Challenging? (2) Untextured regions (Matching ambiguities) There needs to be a certain amount of intensity/color variation (i.e. texture) so that a pixel can be uniquely matched in the other view. Can you (as a human) depict depth if you are standing in front of a wall that is completely white?

    51. Why is Stereo Matching Challenging? (3) Occlusion Problem There are pixels that are only visible in exactly one view. We call this pixels occluded (or half-occluded) It is difficult to estimate depth for these pixels. Occlusion problem makes stereo more challenging than a lot of other computer vision problems.

    52. The Occlusion Problem Lets consider a simple scene composed of a foreground and a background object

    53. Regular case: The white pixel P1 can be seen by both camera.

    54. Occlusion in the right camera: The left camera sees the grey pixel P2. The ray from the right camera to P2 hits the white foreground object => P2 cannot be seen by right camera.

    55. Occlusion in the left camera: The right camera sees the grey pixel P3. The ray from the left camera to P3 hits the white foreground object => P3 cannot be seen by left camera.

    56. Occlusions occur in the proximity of disparity discontinuities.

    57. Occlusions occur in the proximity of disparity discontinuities.

    58. In the left image, occlusions are located to the left of a disparity boundary. In the right image, occlusions are located to the right of a disparity boundary.

    59. It is difficult to find disparity in the matching point does not exist. Ignoring the occlusion problem leads to disparity artefacts near disparity borders.

    60. It is difficult to find disparity in the matching point does not exist. Ignoring the occlusion problem leads to disparity artefacts near disparity borders.

    61. Michael Bleyer LVA Stereo Vision

    62. Assumptions Assumptions are needed to solve the stereo matching problem. Stereo methods differ in What assumptions they use How they implement these assumptions We have already learned two assumptions: Which ones?

    63. Photo Consistency and Epipolar Assumptions Photo consistency assumption: Corresponding pixels have the same intensity/color in both images. Epipolar assumption: The matching point of a pixel has to lie on the same horizontal scanline in the other image. We can combine both assumptions to obtain our first stereo algorithm. Algorithm 1: For each pixel p of the left image, search the pixel q in the right image that lies on the same y-coordinate as p (Epipolar assumption) and has the most similar color in comparison to p (Photo Consistency).

    64. Results of Algorithm 1 Quite disappointing, why? We have posed the following task: I have a red pixel. Find me a red pixel in the other image. Problem: There are usually many red pixels in the other image (ambiguity) We need additional assumptions.

    65. Results of Algorithm 1 What is the most obvious difference between the correct and the computed disparity maps?

    66. Smoothness Assumption (1) Observation: A correct disparity map typically consists of regions of constant (or very similar) disparity. For example, lamp, head, table, We can give this apriori knowledge to a stereo algorithm in the form of a smoothness assumption

    67. Smoothness Assumption (2) Smoothness assumption: Spatially close pixels have the same (or similar) disparity. (By spatially close I mean pixels of similar image coordinates.) Smoothness assumption typically holds true almost everywhere, except at disparity borders.

    68. Smoothness Assumption (3) Almost every stereo algorithm uses the smoothness assumption. Stereo algorithms are commonly divided into two categories based on the form in which they apply the smoothness assumption. These categories are: Local methods Global methods

    69. Local Methods Compare small windows in left and right images. Within the window, pixels are supposed to have the same disparity => implicit smoothness assumption. We will learn a lot about them in the next session.

    70. Global Methods Define a cost function to measure the quality of a disparity map: High costs mean that the disparity map is bad. Low costs mean it is good. Costs function is typically in the form of: where Edata measures photo consistency Esmooth measures smoothness Global methods express smoothness assumption in an explicit form (as a smoothness term). The challenge is to find a disparity map of minimum costs (sessions 4 and 5).

    71. Uniqueness Constraint The uniqueness constraint will help us to handle the occlusion problem. It states: A pixel in one frame has at most a single matching point in the other frame. In general valid, but broken for: Transparent objects Slanted surfaces

    72. Uniqueness Constraint

    73. Uniqueness Constraint

    74. Uniqueness Constraint

    75. Uniqueness Constraint

    76. Other Assumptions Ordering assumption: The order in which pixels occur is preserved in both images. Does not hold for thin foreground object. Disparity gradient limit: Originates from psychology Not clear whether assumption is valid for arbitrary camera setups. Both assumptions have rarely been used recently => they are slightly obsolete.

    77. Michael Bleyer LVA Stereo Vision

    78. Ground Truth Data Ground truth = the correct solution to a given problem. The absence of ground truth data has represented a major problem in computer vision: For most computer vision problems, not a single real test image with ground truth solution has been available. Computer-generated ground truth images do oftentimes not reflect the challenges of real data recorded with a camera. It is difficult to measure the progress in a field if there is no commonly agreed data set with ground truth solution.

    79. Ground Truth Data Ground Truth data is now available for a wide range of computer vision problems including: Object recognition Alpha matting Optical flow MRF-optimization Multi view reconstruction For stereo, ground truth data is available on the Middlebury Stereo Evaluation website http://vision.middlebury.edu/stereo/ The Middlebury set is widely adopted in the stereo community.

    80. The Middlebury Set

    81. How Can One Generate Ground Truth Disparities? Hand labelling: Tsukuba test set Extremely labor-intensive Most other Middlebury ground truth disparity maps have been created using a more precise depth computation technique than stereo matching, namely structured light.

    82. Setup Used for Generating the Middlebury Images Different light patterns are projected onto the scene to compute a high-quality depth map (Depth from structured light).

    83. Setup Used for Generating the Middlebury Images Different light patterns are projected onto the scene to compute a high-quality depth map (Depth from structured light).

    84. Disparity Map Quality Evaluation in the Middlebury Benchmark Estimation of wrong pixels: Build absolute difference between computed and ground truth disparity maps. If absolute disparity difference is larger than one pixel, pixel is counted as error. 3 Error metrics: Percentage of erroneous pixels in (1) unoccluded regions, (2) the whole image and (3) in regions close to disparity borders.

    85. The Middlebury Online Benchmark If you have implemented a stereo algorithm, you can evaluate its performance using the Middlebury benchmark. You have to run it on these 4 image pairs: The 3 error metrics are then computed for each image pair. Your algorithm is then ranked according to the computed error values.

    86. The Middlebury Table Currently, more than 70 methods evaluated. You should use this table to rank your stereo matching algorithm developed as your home work.

    87. General Findings in the Middlebury Table Global methods outperform local methods. Local methods: Adaptive weight methods represent the state-of-the-art. Global methods: Methods that apply Belief Propagation or Graph-Cuts in the optimization step outperform dynamic programming methods (if such categorization makes sense) All top-performing methods apply color segmentation.

    88. Summary 3D geometry Challenges Ambiguity Occlusions Assumptions Photo consistency Smoothness assumption Uniqueness assumption Middlebury benchmark

More Related