Multiple View Geometry in Computer Vision

1. Multiple View Geometryin Computer Vision Marc Pollefeys Comp 290-089

2. Multiple View Geometry A a a c c b b f(a,b,c)=0 (a,b) A (reconstruction) (a,b,c) (a,b,c) (calibration) (a,b) c (transfer)

3. Course objectives • To understand the geometric relations between multiple views of scenes. • To understand the general principles of parameter estimation. • To be able to compute scene and camera properties from real world images using state-of-the-art algorithms.

4. Relation to other vision/image courses • Focuses on geometric aspects • No image processing • Comp 254: Image Processing an Analysis Mostly orthogonal to this course, complementary • Comp 256: Computer Vision (fall 2003) Will be much broader, based on new book: “Computer Vision: a modern approach” David Forsyth and Jean Ponce

5. Material Textbook: Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman Cambridge University Press Alternative book: The Geometry from Multiple Images by Olivier Faugeras and Quan-Tuan Luong MIT Press On-line tutorial: http://www.cs.unc.edu/~marc/tutorial.pdf http://www.cs.unc.edu/~marc/tutorial/

6. Learning approach • read the relevant chapters of the books and/or reading assignements before the course.   • In the course the material will then be covered in detail and motivated with real world examples and applications.  • Small hands-on assignements will be provided to give students a "feel" of the practical aspects. • Students will also read and present some seminal papers to provide a complementary view on some of the covered topics. • Finally, there will also be a project where students will implement an algorithm or approach using concepts covered by the course.    Grade distribution • Class participation: 20% • Hands-on assignments: 10% • Paper presentation: 10% • Implementation assignment/project: 40% • Final: 20%

7. Applications • MatchMoving Compute camera motion from video (to register real an virtual object motion)

8. Applications • 3D modeling

9. Content • Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. • Single View: Camera model, Calibration, Single View Geometry. • Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. • Three Views: Trifocal Tensor, Computing T. • More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality

10. Multiple View Geometry course schedule(tentative)

11. Fast Forward! • Quick overview of what is coming…

12. Background La reproduction interdite (Reproduction Prohibited), 1937, René Magritte.

13. Projective 2D Geometry • Points, lines & conics • Transformations • Cross-ratio and invariants

14. Projective 3D Geometry • Points, lines, planes and quadrics • Transformations • П∞, ω∞and Ω ∞

15. Estimation How to compute a geometric relation from correspondences, e.g. 2D trafo • Linear (normalized), non-linear and Maximum Likelihood Estimation • Robust (RANSAC)

16. Evaluation and error analysis How good are the results we get • Bounds on performance • Covariance propagation & Monte-Carlo estimation error residual

17. Single-View Geometry The Cyclops, c. 1914, Odilon Redon

18. Camera Models Mostly pinhole camera model but also affine cameras, pushbroom camera, …

19. Camera Calibration • Compute P given (m,M) (normalized) linear, MLE,… • Radial distortion

20. More Single-View Geometry • Projective cameras and planes, lines, conics and quadrics. • Camera center and camera rotation • Camera calibration and vanishing points, calibrating conic and the IAC

21. Single View Metrology Antonio Criminisi

22. Two-View Geometry The Birth of Venus (detail), c. 1485, Sandro Botticelli

23. Epipolar Geometry Fundamental matrix Essential matrix

24. Two-View Reconstruction

25. Epipolar Geometry Computation (normalized) linear: minimal: MLE: RANSAC … and automated two view matching

26. Rectification Warp images to simplify epipolar geometry

27. Structure Computation • Points: Linear, optimal, direct optimal • Also lines and vanishing points

28. Planes and Homographies Relation between plane and H given P and P’ Relation between H and F, H from F, F from H The infinity homography H∞

29. Three-View Geometry The Birth of Venus (detail), c. 1485, Sandro Botticelli

30. Trifocal Tensor

31. Three View Reconstruction • (normalized) linear • minimal (6 points) • MLE (Gold Standard)

32. Multiple-View Geometry The Birth of Venus (detail), c. 1485, Sandro Botticelli

33. Multiple View Geometry Quadrifocal tensor 81 parameters, but only 29 DOF!

34. Multiple View Reconstruction • Affine factorization • Projective factorization

35. Multiple View Reconstruction • Sequential reconstruction

36. P1 P2 P3 M U1 U2 W U3 WT V 3xn (in general much larger) 12xm Bundle Adjustment Maximum Likelyhood Estimation for complete structure and motion

37. U-WV-1WT WT V 11xm 3xn Bundle Adjustment Maximum Likelyhood Estimation for complete structure and motion

38. Bundle adjustment No bundle adjustment Bundle adjustment needed to avoid drift of virtual object throughout sequence Bundle adjustment (including radial distortion)

39. * * projection constraints Auto-calibration

40. Dynamic Structure from Motion

41. Cheirality Oriented projective geometry • Allows to use fact that points are in front of camera • to recover quasi-affine reconstruction • to determine order for image warping • to determine orientation for rectification • with epipoles in images • etc.

42. Duality Gives possibility to interchange role of P and X in algorithms

43. Contact information Marc Pollefeys, Room 205 marc@cs.unc.edu Tel. 962 1845