1 / 73

3-D Computer Vision CSc 83029

3-D Computer Vision CSc 83029. Stereo. Stereopsis. Recovering 3D information (depth) from two images. The correspondence problem. The reconstruction problem. Epipolar constraint. The 8-point algorithm. The 2 problems of Stereo. The setting: Simultaneous acquisition of 2 images

bell
Download Presentation

3-D Computer Vision CSc 83029

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3-D Computer VisionCSc 83029 Stereo CSc83029 3-D Computer Vision / Ioannis Stamos

  2. Stereopsis • Recovering 3D information (depth) from two images. • The correspondence problem. • The reconstruction problem. • Epipolar constraint. • The 8-point algorithm. CSc83029 3-D Computer Vision / Ioannis Stamos

  3. The 2 problems of Stereo The setting: Simultaneous acquisition of 2 images (left, right) of a static scene. • Correspondence: Which parts of the left and right images are projections of the same scene element? • Reconstruction: Given: • A number of corresponding points between the left and right image, • Information on the geometry of the stereo system, Find: 3-D structure of observed objects. CSc83029 3-D Computer Vision / Ioannis Stamos

  4. Stereo Vision depth map CSc83029 3-D Computer Vision / Ioannis Stamos

  5. A simple stereo system Fixation Point:Infinity. Parallel optical axes. P Z-axis Z cl cr f X-axis Or Ol T Left Camera Right Camera CSc83029 3-D Computer Vision / Ioannis Stamos

  6. Triangulation Fixation Point:Infinity. Parallel optical axes. P Z-axis Z cl cr pl pr f X-axis Or Ol T Left Camera Right Camera Calibrated Cameras CSc83029 3-D Computer Vision / Ioannis Stamos

  7. Triangulation Fixation Point:Infinity. Parallel optical axes. P Z-axis xl Z xr cl cr pl pr f X-axis Or Ol T Left Camera Right Camera Calibrated Cameras Similar triangles: d:disparity (difference in retinal positions). T:baseline. Depth (Z) is inversely proportional to d (fixation at infinity)

  8. Triangulation Fixation Point:Infinity. Parallel optical axes. P Z-axis xl Z xr cl cr pl pr f X-axis Or Ol T Left Camera Right Camera Calibrated Cameras Similar triangles: d:disparity (difference in retinal positions). T:baseline. Baseline T: accuracy/robustness of depth calculation.

  9. Triangulation Fixation Point:Infinity. Parallel optical axes. P Z-axis xl Z xr cl cr pl pr f X-axis Or Ol T Left Camera Right Camera Calibrated Cameras Similar triangles: d:disparity (difference in retinal positions). T:baseline. Small baselines: less accurate measurements.

  10. Triangulation Fixation Point:Infinity. Parallel optical axes. P Z-axis xl Z xr cl cr pl pr f X-axis Or Ol T Left Camera Right Camera Calibrated Cameras Similar triangles: d:disparity (difference in retinal positions). T:baseline. Large baselines: occlusions/foreshortening.

  11. Parameters of Stereo System Fixation Point:Infinity. Parallel optical axes. P Z-axis xl Z xr cl cr pl pr f X-axis Or Ol T Left Camera Right Camera • Intrinsic parameters (i.e. f, cl, cr) • Extrinsic parameters: relative position and • orientation of the 2 cameras. STEREO CALIBRATION PROBLEM CSc83029 3-D Computer Vision / Ioannis Stamos

  12. Stereo – Photometric Constraint • Same world point has same intensity in both images. • Lambertian fronto-parallel • Issues (noise, specularities, foreshortening) From Jana Kosecka • Difficulties – ambiguities, large changes of appearance, due to change • Of viewpoint, non-uniquess

  13. Correspondence Is Difficult Ambiguity: there may be many possible 3D reconstructions. CSc83029 3-D Computer Vision / Ioannis Stamos

  14. Correspondence Is Difficult No texture: difficult to find a unique match. CSc83029 3-D Computer Vision / Ioannis Stamos

  15. Correspondence Is Difficult Foreshortening: the projection in each image is different. CSc83029 3-D Computer Vision / Ioannis Stamos

  16. Correspondence Is Difficult Occlusions: there may not be a correspondence. Assumptions: 1) Most scene points are visible from both views. 2) Corresponding image regions are similar.

  17. Correspondence Is Difficult Curved surfaces: triangulation produces incorrect position. CSc83029 3-D Computer Vision / Ioannis Stamos

  18. Correspondence is difficult: The Ordering Constraint Points appear in the same order But it is not always the case ... CSc83029 3-D Computer Vision / Ioannis Stamos

  19. More Correspondence Problems • Regions without texture • Highly Specular surfaces • Translucent objects CSc83029 3-D Computer Vision / Ioannis Stamos

  20. Methods For Correspondence • Correlation based (dense correspondences). • Feature based (such as edges/lines/corners). CSc83029 3-D Computer Vision / Ioannis Stamos

  21. Correlation-Based Methods R(pl ) pl Left Image Right Image 1) For each pixel pl in the left image search in a region R(pl) in the right image for corresponding pixel pr. 2) Use image windows of size (2W+1)x(2W+1). 3) Select the pixel pr that maximizes a correlation function. HAVE TO SPECIFY: Region R, size W, and correlation function ψ.

  22. Correlation-Based Methods R(pl ) pl Left Image Right Image For each pixel pl=[i,j] in the left image For each displacement d=[d1,d2] in R(pl) Compute The disparity of pl is the d that maximizes c(d) HAVE TO SPECIFY: Region R, size W, and correlation function ψ.

  23. Correlation-Based Methods R(pl ) pl Left Image Right Image CROSS-CORRELATION SUM OF SQUARED DIFFERENCES SSD SSD is usually preferred: handles different intensity scales. Normalized cross-correlation is better (but is more expensive).

  24. Correspondence Is Difficult Intensities in window may differ. Normalized cross-correlation may help. CSc83029 3-D Computer Vision / Ioannis Stamos

  25. Image Normalization • Even when the cameras are identical models, there can be differences in gain and sensitivity. • The cameras do not see exactly the same surfaces, so their overall light levels can differ. • For these reasons and more, it is a good idea to normalize the pixels in each window: CSc83029 3-D Computer Vision / Ioannis Stamos From Sebastian Thrun/Jana Kosecka

  26. Comparing Windows: Minimize Sum of Squared Differences Maximize Cross correlation It is closely related to the SSD: CSc83029 3-D Computer Vision / Ioannis Stamos From Jana Kosecka

  27. Region based Similarity Metrics • Sum of squared differences • Normalize cross-correlation • Sum of absolute differences CSc83029 3-D Computer Vision / Ioannis Stamos From Jana Kosecka

  28. NCC score for two widely separated views NCC score CSc83029 3-D Computer Vision / Ioannis Stamos From Jana Kosecka

  29. W = 3 W = 20 Window size • Effect of window size • Better results with adaptive window • T. Kanade and M. Okutomi,A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991. • D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 (S. Seitz) CSc83029 3-D Computer Vision / Ioannis Stamos

  30. Stereo results • Data from University of Tsukuba Scene Ground truth (Seitz) CSc83029 3-D Computer Vision / Ioannis Stamos

  31. Results with window correlation Window-based matching (best window size) Ground truth (Seitz) CSc83029 3-D Computer Vision / Ioannis Stamos

  32. Results with better method State of the art Ground truth • Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, • International Conference on Computer Vision, September 1999. (Seitz) CSc83029 3-D Computer Vision / Ioannis Stamos

  33. Feature-Based Methods Left Image Right Image Match sparse sets of extracted features. A feature descriptor for a line could contain: length l, orientation o, midpoint (x,y), average contrast c An example similarity measure (w’s are weights): CSc83029 3-D Computer Vision / Ioannis Stamos

  34. Correspondence Using Correlation Left Disparity Map Images courtesy of Point Grey Research CSc83029 3-D Computer Vision / Ioannis Stamos

  35. LEFT IMAGE line corner structure Correspondence By Features CSc83029 3-D Computer Vision / Ioannis Stamos From Sebastian Thrun/Jana Kosecka

  36. Correspondence By Features line corner structure RIGHT IMAGE • Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum From Sebastian Thrun/Jana Kosecka

  37. Dense depth maps. Need textured images Sensitive to foreshorening/illumination changes Need close views Sparse depth maps. Insensitive to illumination changes. A-priori info used. Faster. Comparison of Matching Methods Correlation-Based Feature-Based Problems: occlusions/spurious matches: =>Introduce constraints in matching (i.e. left-right consistency constraint) CSc83029 3-D Computer Vision / Ioannis Stamos

  38. Epipolar Constraint (Geometry) Scene point P Pl Pr EPIPOLARLINE EPIPOLARLINE EPIPOLARPLANE Image plane πr Image plane πl pr pl el er Ol Or Center of projection Center of projection Epipoles CSc83029 3-D Computer Vision / Ioannis Stamos

  39. Epipolar Constraint Scene point P Pl Pr EPIPOLARLINE EPIPOLARLINE EPIPOLARPLANE Image plane πr Image plane πl pr pl el er Ol Or Center of projection Center of projection Epipoles Extrinsic parameters: Left/Right Camera Frames: Pr=R(Pl-T), T=Or-Ol (1)

  40. Epipolar Constraint Ol, Or, pl => Enough to define right E.L. Scene point P Pl Pr EPIPOLARLINE EPIPOLARLINE EPIPOLARPLANE Image plane πr Image plane πl pr pl el er Ol Or Center of projection Center of projection Epipoles Given pl, pr is constrained to lie on the Epipolar Line (E.L.). For each left pixel pl, find the corresponding right E.L. Searching for pr reduces to a 1-D problem.

  41. Epipolar Constraint Scene points Image plane el er Center of projection Center of projection Epipoles All E.L.s go through epipoles. Parallel image planes => epipoles at infinity.

  42. Essential Matrix Estimate the epipolar geometry: correspondence between points and E.L.s. (1) Link bw/ epipolar constraint and extrinsic parameters of stereo system.

  43. Essential Matrix Perspective: pl=[xl,yl,fl]T, pr=[xr,yr,zr]T pl= fl/ZlPl, pr=fr/Zr Pr Perspective Projection el er Essential matrix Rank 2 Epipolar lines are found by CSc83029 3-D Computer Vision / Ioannis Stamos

  44. Camera Models (linear versions) Elegant decomposition. No distortion! Homogeneous Coordinates Measured Pixel (xim, yim) World Point (Xw, Yw,Zw) ?

  45. Fundamental Matrix Ml (Mr) matrix of intrinsic parameters for left (right) camera. Camera to pixel coordinates: Essential matrix equation becomes: el er Fundamental matrix F: pixel coordinates ! E: camera coordinates ! Epipolar lines: CSc83029 3-D Computer Vision / Ioannis Stamos

  46. Encodes information on extrinsic parameters. Has rank 2. Its 2 non-zero singular values are equal. Encodes information on both the extrinsic and intrinsic parameters. Has rank 2. Conclusions Essential Matrix Fundamental Matrix CSc83029 3-D Computer Vision / Ioannis Stamos

  47. el er Estimating the epipolar geometry el er CSc83029 3-D Computer Vision / Ioannis Stamos

  48. el er Estimating the epipolar geometry el er Problem: Find the fundamental matrix from a set of image correspondences CSc83029 3-D Computer Vision / Ioannis Stamos

  49. el er Estimating the epipolar geometry With the respect to the constraint: Rank(F) = 2. CSc83029 3-D Computer Vision / Ioannis Stamos

  50. The 8-point algorithm n>=8 correspondences v: the 9 elements of F. A: n x 9 measurement matrix. Solve using SVD (solution up to a scale factor). Enforce rank(F)=2 =>SVD on the computed F. Be careful: numerical instabilities.

More Related