Stereo and Epipolar geometry

1. Stereo and Epipolar geometry Jana Kosecka CS223b

2. Previously • Image Primitives (feature points, lines, contours) • Today: • How to match primitives between two (multiple) views) • Goals: 3D reconstruction, recognition • Stereo matching and reconstruction (canonical configuration) • Epipolar Geometry (general two view setting) CS223b

3. Why Stereo Vision? • 2D images project 3D points into 2D: P Q P’=Q’ O center of projection • 3D points on the same viewing line have the same 2D image: • 2D imaging results in depth information loss CS223b

4. Canonical Stereo Configuration • Assumes (two) cameras • Known positions and focal lengths • Recover depth P f O1 O2 T CS223b

5. Depth can be recovered by triangulation • Prerequisite: Finding Correspondences Given an image point in left image, the (corresponding) point in the right image, is the point which is the projection of the same 3-D point CS223b

6. Random dot stereograms B. Julesz: Showed that correspondence is not needed for stereo. CS223b

7. Autostereograms • Depth perception from one image • Viewing trick the brain by focusing at the plane behind - match can be established perception of 3D CS223b

8. Correspondence Problem • Two classes of algorithms: • Correlation-based algorithms • Produce a DENSE set of correspondences • Feature-based algorithms • Produce a SPARSE set of correspondences CS223b

9. Stereo – Photometric Constraint • Same world point has same intensity in both images. • Lambertian fronto-parallel • Issues (noise, specularities, foreshortening) • Difficulties – ambiguities, large changes of appearance, due to change • Of viewpoint, non-uniquess CS223b

10. Stereo Matching What if ? For each scanline , for each pixel in the left image • compare with every pixel on same epipolar line in right image • pick pixel with minimum match cost • This will never work, so: improvement match windows CS223b (slides O. Camps)

11. ? = g f Most popular Comparing Windows: For each window, match to closest window on epipolar line in other image. (slides O. Camps) CS223b

12. Comparing Windows: Minimize Sum of Squared Differences Maximize Cross correlation It is closely related to the SSD: CS223b

13. Region based Similarity Metrics • Sum of squared differences • Normalize cross-correlation • Sum of absolute differences CS223b

14. NCC score for two widely separated views NCC score CS223b

15. W = 3 W = 20 Window size • Effect of window size • Better results with adaptive window • T. Kanade and M. Okutomi,A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991. • D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 (S. Seitz) CS223b

16. Stereo results • Data from University of Tsukuba Scene Ground truth (Seitz) CS223b

17. Results with window correlation Window-based matching (best window size) Ground truth (Seitz) CS223b

18. Results with better method State of the art Ground truth • Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, • International Conference on Computer Vision, September 1999. (Seitz) CS223b

19. More of advanced stereo • Ordering constraint • Dynamic programming • Global optimization – next lecture CS223b

20. Two view – General Configuration • Motion between the two views is not known Given two views of the scene recover the unknown camera displacement and 3D scene structure CS223b

21. Pinhole Camera Imaging Model • 3D points • Image points • Perspective Projection • Rigid Body Motion • Rigid Body Motion + Persp. projection CS223b

22. Rigid Body Motion – Two Views CS223b

23. 3D Structure and Motion Recovery Measurements Corresponding points unknowns Find such Rotation and Translation and Depth that the reprojection error is minimized Euclidean transformation Two views ~ 200 points 6 unknowns – Motion 3 Rotation, 3 Translation - Structure 200x3 coordinates - (-) universal scale Difficult optimization problem CS223b

24. Notation • Cross product between two vectors in where CS223b

25. Epipolar Geometry Image correspondences • Algebraic Elimination of Depth [Longuet-Higgins ’81]: • Essential matrix CS223b

26. Epipolar Geometry • Epipolar lines • Epipoles Image correspondences Epipolar transfer • Additional constraints CS223b

27. Characterization of Essential Matrix Essential matrix special 3x3 matrix (Essential Matrix Characterization) A non-zero matrix is an essential matrix iff its SVD: satisfies: with and and CS223b

28. Estimating Essential Matrix • Find such Rotation and Translation that the epipolar error is minimized • Space of all Essential Matrices is 5 dimensional • 3 DOF Rotation, 2 DOF – Translation (up to scale !) • Denote • Rewrite • Collect constraints from all points CS223b

29. Estimating Essential Matrix • Solution • Eigenvector associated with the smallest eigenvalue of • If degenerate configuration estimated using linear least squares unstack -> Projection on to Essential Space (Project onto a space of Essential Matrices) If the SVD of a matrix is given by then the essential matrix which minimizes the Frobenius distance is given by with CS223b

30. Pose Recovery from Essential Matrix Essential matrix (Pose Recovery) There are two relative poses with and corresponding to a non-zero matrix essential matrix. • Twisted pair ambiguity CS223b

31. Two view linear algorithm - summary • Solve the LLSEproblem: • Solution eigenvector associated with • smallest eigenvalue of • Compute SVD of F recovered from data E is 5 diml. sub. mnfld. in • 8-point linear algorithm • Projectonto the essential manifold: • Recover the unknown pose: CS223b

32. Pose Recovery • There are two pairs corresponding to essential matrix . • There are two pairs corresponding to essential matrix . • Positive depth constraint disambiguates the impossible solutions • Translation has to be non-zero • Points have to be in general position • - degenerate configurations – planar points • - quadratic surface • Linear 8-point algorithm • Nonlinear 5-point algorithms yields up to 10 solutions CS223b

33. 3D Structure Recovery unknowns • Eliminate one of the scale’s • Solve LLSE problem • Alternatively recover each point depth separately If the configuration is non-critical, the Euclidean structure of the points and motion of the camera can be reconstructed up to a universal scale. CS223b

34. Example Two views Point Feature Matching CS223b

35. Example Epipolar Geometry Camera Pose and Sparse Structure Recovery CS223b

36. Epipolar Geometry - Planar case Image correspondences • Plane in first camera coordinate frame Planar homography Linear mapping relating two corresponding planar points in two views CS223b

37. Decomposition of H (into motion and plane normal) • Algebraic elimination of depth • can be estimated linearly • Normalization of • Decomposition of H into 4 solutions CS223b

38. calibrated coordinates Linear transformation pixel coordinates Uncalibrated Camera CS223b

39. Uncalibrated Epipolar Geometry • Epipolar constraint • Fundamental matrix CS223b

40. Properties of the Fundamental Matrix • Epipolar lines • Epipoles Image correspondences CS223b

41. f f Or Ol T el er Epipolar Geometry for Parallel Cameras P Epipoles are at infinite Epipolar lines are parallel to the baseline CS223b

42. Properties of the Fundamental Matrix A nonzero matrix is a fundamental matrix if has a singular value decomposition (SVD) with for some . There is little structure in the matrix except that CS223b

43. Estimating Fundamental Matrix • Find such Fthat the epipolar error is minimized Pixel coordinates • Fundamental matrix can be estimated up to scale • Denote • Rewrite • Collect constraints from all points CS223b

44. Two view linear algorithm – 8-point algorithm • Solve the LLSEproblem: • Solution eigenvector associated with • smallest eigenvalue of • Compute SVD of F recovered from data • Projectonto the essential manifold: • cannot be unambiguously decomposed into pose • and calibration CS223b

45. Projective Reconstruction • Euclidean Motion Cannot be obtained in uncalibrated setting (F cannot be uniquely decomposed into R,T and K matrix) • Can we still say something about 3D ? • Notion of the projective 3D structure (study of projective geometry) Euclidean reconstruction Projective reconstruction CS223b

46. Euclidean vs Projective reconstruction • Euclidean reconstruction – true metric properties of objects lenghts (distances), angles, parallelism are preserved • Unchanged under rigid body transformations • => Euclidean Geometry – properties of rigid bodies under rigid body transformations, similarity transformation • Projective reconstruction – lengths, angles, parallelism are NOT preserved – we get distorted images of objects – their distorted 3D counterparts --> 3D projective reconstruction • => Projective Geometry CS223b

47. Image rectification • Given general displacement how to warp the views • Such that epipolar lines are parallel to each other • How to warp it back to canonical configuration (more details later) (Seitz) CS223b

48. Epipolar rectification • Rectified Image Pair • Corresponding epipolar lines are aligned with the scan-lines • Search for dense correspondence is a 1D search CS223b

49. Epipolar rectification Rectified Image Pair CS223b

50. Special Motions – Pure Rotation • Calibrated Two views related by rotation only • Uncalibrated Case • Mapping to a reference view – H can be estimated • Mapping to a cylindrical surface - applications – image mosaics CS223b