Assignment 2

1. Assignment 2 (due by Wednesday 19/03/03) Compute F automatically from image pair (putative matches, 8-point, 7-point, iterative, RANSAC, guided matching)

2. Rectification and structure computationclass 15 Multiple View Geometry Comp 290-089 Marc Pollefeys

3. Multiple View Geometry course schedule(subject to change)

4. Two-view geometry Epipolar geometry 3D reconstruction F-matrix comp. Structure comp.

5. Automatic computation of F • Interest points • Putative correspondences • RANSAC • (iv) Non-linear re-estimation of F • Guided matching • (repeat (iv) and (v) until stable)

6. Feature points Select strongest features (e.g. 1000/image)

7. ? Feature matching Evaluate ZNCC,SSD,SAD for all features with similar coordinates Keep mutual best matches Still many wrong matches!

8. (generate hypothesis) (verify hypothesis) RANSAC Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until (#inliers,#samples)<95% Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches

9. Finding more matches: guided matching restrict search range to neighborhood of epipolar line (1.5 pixels) relax disparity restriction (along epipolar line)

10. geometric relations between two views is fully described by recovered 3x3 matrix F two-view geometry

11. e Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

12. ~ image size (calibrated) Planar rectification (standard approach) Distortion minimization (uncalibrated) Bring two views to standard stereo setup (moves epipole to ) (not possible when in/close to image)

15. Polar rectification (Pollefeys et al. ICCV’99) Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Works for all relative motions Guarantees minimal image size

16. polar rectification: example

17. polar rectification: example

18. Example: Béguinage of Leuven Does not work with standard Homography-based approaches

19. Example: Béguinage of Leuven

20. Stereo matching • attempt to match every pixel • use additional constraints

21. Exploiting motion and scene constraints • Epipolar constraint (through rectification) • Ordering constraint • Uniqueness constraint • Disparity limit • Disparity continuity constraint

22. Ordering constraint surface slice surface as a path 6 5 occlusion left 4 3 2 1 4,5 6 1 2,3 5 6 2,3 4 occlusion right 1 3 6 1 2 4,5

23. Uniqueness constraint • In an image pair each pixel has at most one corresponding pixel • In general one corresponding pixel • In case of occlusion there is none

24. use reconsructed features to determine bounding box Disparity constraint surface slice surface as a path bounding box disparity band constant disparity surfaces

25. Disparity continuity constraint • Assume piecewise continuous surface • piecewise continuous disparity • In general disparity changes continuously • discontinuities at occluding boundaries

26. Similarity measure (SSD or NCC) Optimal path (dynamic programming ) Stereo matching • Constraints • epipolar • ordering • uniqueness • disparity limit • disparity gradient limit • Trade-off • Matching cost (data) • Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

27. Hierarchical stereo matching Allows faster computation Deals with large disparity ranges Downsampling (Gaussian pyramid) Disparity propagation (Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

28. Disparity map image I´(x´,y´) image I(x,y) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

29. Example: reconstruct image from neighboring images

30. Multi-view depth fusion (Koch, Pollefeys and Van Gool. ECCV‘98) • Compute depth for every pixel of reference image • Triangulation • Use multiple views • Up- and down sequence • Use Kalman filter Allows to compute robust texture

31. Point reconstruction

32. linear triangulation homogeneous invariance? algebraic error yes, constraint no (except for affine) inhomogeneous

33. geometric error possibility to compute using LM (for 2 or more points) or directly (for 2 points)

34. Geometric error Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution (see Hartley&Sturm,CVIU´97)

35. x1 l1 l1 l2 x1 x2 x2´ x1´ x2 l2 Optimal 3D point in epipolar plane Given an epipolar plane, find best 3D point for (x1,x2) Select closest points (x1´,x2´) on epipolar lines Obtain 3D point through exact triangulation Guarantees minimal reprojection error (given this epipolar plane)

36. m1 l2(a) l1(a) m2 Optimal epipolar plane • Reconstruct matches in projective frame by minimizing the reprojection error • Non-iterative method Determine the epipolar plane for reconstruction Reconstruct optimal point from selected epipolar plane 3DOF (Hartley and Sturm, CVIU´97) 1DOF (polynomial of degree 6 check all minima, incl ∞)

37. Reconstruction uncertainty consider angle between rays

38. Line reconstruction doesn‘t work for epipolar plane

39. Next class: Scene and plane homographies