Multiple View Geometry

1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. Outline • Stereo matching • Self-calibration • Chapter 11 and 18 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. Stereo matching • attempt to match every pixel • use additional constraints

4. Exploiting motion and scene constraints • Ordering constraint • Uniqueness constraint • Disparity limit • Disparity continuity constraint • Epipolar constraint (through rectification)

5. Ordering constraint surface slice surface as a path 6 5 occlusion left 4 3 2 1 4,5 6 1 2,3 5 6 2,3 4 occlusion right 1 3 6 1 2 4,5

6. Uniqueness constraint • In an image pair each pixel has at most one corresponding pixel • In general one corresponding pixel • In case of occlusion there is none

7. use reconsructed features to determine bounding box Disparity constraint surface slice surface as a path bounding box disparity band constant disparity surfaces

8. Disparity continuity constraint • Assume piecewise continuous surface • piecewise continuous disparity • In general disparity changes continuously • discontinuities at occluding boundaries

9. Similarity measure (SSD or NCC) Optimal path (dynamic programming ) Stereo matching • Constraints • epipolar • ordering • uniqueness • disparity limit • disparity gradient limit • Trade-off • Matching cost (data) • Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

10. Hierarchical stereo matching Allows faster computation Deals with large disparity ranges Downsampling (Gaussian pyramid) Disparity propagation (Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

11. Disparity map image I´(x´,y´) image I(x,y) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

12. Example: reconstruct image from neighboring images

13. Multi-view depth fusion (Koch, Pollefeys and Van Gool. ECCV‘98) • Compute depth for every pixel of reference image • Triangulation • Use multiple views • Up- and down sequence • Use Kalman filter Allows to compute robust texture

14. Point reconstruction

15. Linear triangulation homogeneous invariance? Inhomogeneous is affine invariant inhomogeneous

16. Geometric error Can be compute using Levenberg-Marquadt (for 2 or more points) or directly (for 2 points)

17. Geometric error Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution (see Hartley&Sturm,CVIU´97)

18. Reconstruction uncertainty consider angle between rays

19. Outline • Stereo matching • Self-calibration • Introduction • Self-calibration • Dual Absolute Quadric • Critical Motion Sequences

20. Motivation • Avoid explicit calibration procedure • Complex procedure • Need for calibration object • Need to maintain calibration

21. Motivation • Allow flexible acquisition • No prior calibration necessary • Possibility to vary intrinsics • Use archive footage

22. Example

23. Projective ambiguity Reconstruction from uncalibrated images  projective ambiguity on reconstruction

24. Stratification of geometry Projective Affine Metric 15 DOF 7 DOF absolute conic angles, rel.dist. 12 DOF plane at infinity parallelism More general More structure

25. Constraints ? Scene constraints • Parallellism, vanishing points, horizon, ... • Distances, positions, angles, ... Unknown scene  no constraints • Camera extrinsics constraints • Pose, orientation, ... Unknown camera motion  no constraints • Camera intrinsics constraints • Focal length, principal point, aspect ratio & skew Perspective camera model too general  some constraints

26. Euclidean projection matrix Factorization of Euclidean projection matrix Intrinsics: (camera geometry) (camera motion) Extrinsics: Note: every projection matrix can be factorized, but only meaningful for euclidean projection matrices

27. Constraints on intrinsic parameters Constant e.g. fixed camera: Known e.g. rectangular pixels: square pixels: principal point known:

28. Self-calibration Upgrade from projective structure to metric structure using constraintsonintrinsic camera parameters • Constant intrinsics • Some known intrinsics, others varying • Constraints on intrincs and restricted motion (e.g. pure translation, pure rotation, planar motion) (Faugeras et al. ECCV´92, Hartley´93, Triggs´97, Pollefeys et al. PAMI´99, ...) (Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...) (Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)

29. A counting argument • To go from projective (15DOF) to metric (7DOF) at least 8 constraints are needed • Minimal sequence length should satisfy • Independent of algorithm • Assumes general motion (i.e. not critical)

30. Self-calibration:conceptual algorithm Given projective structure and motion {Pj,Mi}, then the metric structure and motion can be obtained as {PjT-1,TMi}, with cost given constraints function extracting intrinsics from projection matrix

31. quadrics transformations projection Conics & Quadrics conics

32. The Absolute Dual Quadric (Triggs CVPR´97) Degenerate dual quadric * Encodes both absolute conic  and    * for metric frame:

33. Absolute Dual Quadric and Self-calibration Eliminate extrinsics from equation Equivalent to projection of dual quadric Abs.Dual Quadric also exists in projective world Transforming world so that reduces ambiguity to metric

34. * * projection constraints Absolute Dual Quadric and Self-calibration Projection equation: Translate constraints on K through projection equationto constraints on * Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics

35. Constraints on * #constraints condition constraint type

36. Linear algorithm (Pollefeys et al.,ICCV´98/IJCV´99) Assume everything known, except focal length Yields 4 constraint per image Note that rank-3 constraint is not enforced

37. Linear algorithm revisited (Pollefeys et al., ECCV‘02) Weighted linear equations assumptions

38. Projective to metric Compute T from using eigenvalue decomposition of and then obtain metric reconstruction as

39. Alternatives: (Dual) image of absolute conic • Equivalent to Absolute Dual Quadric • Practical when H can be computed first • Pure rotation(Hartley’94, Agapito et al.’98,’99) • Vanishing points, pure translations, modulus constraint, …

40. Note that in the absence of skew the IAC can be more practical than the DIAC!

41. Kruppa equations Limit equations to epipolar geometry Only 2 independent equations per pair But independent of plane at infinity

42. Refinement • Metric bundle adjustment Enforce constraints or priors on intrinsics during minimization (this is „self-calibration“ for photogrammetrist)