Unified Multiple View Geometry Theory and Algorithms

1. Multiple View Geometry Unified Yi Ma (UIUC), Kun Huang (UIUC) and Jana Kosecka (GMU) by Rene Vidal Electrical Engineering & Computer Sciences University of California at Berkeley http://www.eecs.berkeley.edu

2. FORMULATION: camera model and multiple images ALGEBRA: multilinear constraints v.s. rank deficiency condition GEOMETRY: geometric interpretation of rank deficiency condition ALGORITHM: matching, transfer, motion and structure recovery GENERALIZATION: line features, 3-D curves and surfaces

3. FORMULATION - Fundamental Geometric Problem Input: Corresponding images (of point or line) in multiple images. Output: Camera motion, camera calibration, object 3D structure.

4. FORMULATION – Literature Review Multiple view geometry theory • Two views: Longuet-Higgins’81, Huang & Faugeras’89, … • Three views: Spetsakis & Aloimonos’90, Shashua’94, Hartley’94, … • Four views: Triggs’95, Shashua’00, … • Multiple views: Heyden & Astrom’97’98, Ma et. al.’99, … Multiple view geometry algorithms • Euclidean: Maybank’93, Weng, Ahuja & Huang’93, … • Affine: Quan & Kanade’96, … • Projective: Triggs’96, … • Orthographic: Tomasi & Kanade’92, … Recent books on multiple view geometry 1. Multiple view geometry in computer vision, Hartley & Zisserman’00. 2. Geometry of multiple images, Faugeras & Luong’01.

5. surface curve line point practice projective algorithm affine theory Euclidean 2 views algebra 3 views geometry 4 views optimization m views FORMULATION – An Anatomy of Cases (State of the Art)

6. surface curve line point practice projective algorithm affine theory Euclidean rank deficiency 2 views algebra 3 views geometry 4 views optimization m views FORMULATION – A Need for Unification

7. FORMULATION – Pinhole Camera Model Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane

8. FORMULATION – Hat Operator

9. FORMULATION – Multiple View Structure From Motion Given corresponding images of points: recover everything else from equations: “incidental condition” . . .

10. FORMULATION: camera model and multiple images ALGEBRA: multilinear constraints v.s. rank deficiency condition GEOMETRY: geometric interpretation of rank deficiency condition ALGORITHM: matching, transfer, motion and structure recovery GENERALIZATION: line features, 3-D curves and surfaces

11. ALGEBRA – Multilinear Constraints For images of the same 3-D point : is rank deficient (leading to the conventional approach) Multilinear constraints among 2, 3, 4 views

12. ALGEBRA – Rank Deficiency of the Multiple View Matrix WLOG, choose camera frame 1 as the reference Multiple View Matrix Theorem [Rank Deficiency Condition] (generic) (degenerate) Let then and are linearly dependent.

13. ALGEBRA – M Matrix Implies Bilinear Constraints Fact: Given non-zero vectors Hence, we have These constraints are only necessary but NOT sufficient!

14. ALGEBRA – M Matrix Implies Trilinear Constraints Fact: Given non-zero vectors Hence, we have • These constraints are only necessary but NOT sufficient! • However, there is NO further relationship among any 4 views. • Quadrilinear constraints hence do not exist!

15. FORMULATION: camera model and multiple images ALGEBRA: multilinear constraints v.s. rank deficiency condition GEOMETRY:geometric interpretation of rank deficiency condition ALGORITHM: matching, transfer, motion and structure recovery GENERALIZATION: line features, 3-D curves and surfaces

16. GEOMETRY – Uniqueness of Pre-image by Bilinear Constraints “Bilinear means pair-wise coplanar”: except in a rare coplanar case: Rectilinear motion Trifocal plane

17. GEOMETRY – Uniqueness of Pre-image by Trilinear Constraints “Trilinear means triple-wise incidental”: except in a rare collinear case:

18. GEOMETRY – Uniqueness of Pre-image by M Matrix Theorem [Uniqueness of Pre-image] Given vectors with respect to camera frames, they correspond to a unique point in the 3-D space if the rank of the matrix is of rank 1. If the rank is 0, the point is determined up to a line on which all the camera centers must lie. “incidental condition” . . .

19. GEOMETRY – Geometric Interpretation of M Matrix is the “depth” of the point relative to the camera center. Points that give the same matrix are on a sphere of radius

20. FORMULATION: camera model and multiple images ALGEBRA: multilinear constraints v.s. rank deficiency condition GEOMETRY: geometric interpretation of rank deficiency condition ALGORITHM:matching, transfer, motion and structure recovery GENERALIZATION: line features, 3-D curves and surfaces

21. ALGORITHM 1 – Multiple View Matching Test Given the projection matrix associated to camera frames. Then for vectors

22. ALGORITHM 2 – Motion and Structure from Multiple Views Given images of points:

23. ALGORITHM 2 – SVD Based Four Step Algorithm

24. ALGORITHM 2 – Simulation Results Motion XX-YY, 1000 trials and T/R ratio 1.5

25. ALGORITHM 2 – Simulation Results Motion XX-YY-ZZ, 1000 trials and T/R ratio 1.5

26. ALGORITHM 3 – Mapping Images to a New View Given the projection matrix associated to camera frames. Then for given vectors So given images, rank deficiency adds a linear constraint on the image. Computing the kernel of gives the new image.

27. FORMULATION: camera model and multiple images ALGEBRA: multilinear constraints v.s. rank deficiency condition GEOMETRY: geometric interpretation of rank deficiency condition ALGORITHM: matching, transfer, motion and structure recovery GENERALIZATION:line features, 3-D curves and surfaces

28. GENERALIZATION – Line Features Homogeneous representation of a 3-D line Homogeneous representation of its 2-D image Projection of a 3-D line to an image plane

29. GENERALIZATION – Multiple View Matrix: Line v.s. Point Point Features Line Features

30. GENERALIZATION – Point/Point Duality Point/point duality between a camera center and a 3-D point: Theorem [Point/Point Duality] From matrix only, if a camera center is moving on a straight line, a fixed 3-D point is determined up to a circle; if a camera center is fixed but a point is moving on a line, the line is determined up to a circle.

31. GENERALIZATION – Line Features (v.s. Point Features)

32. GENERALIZATION – SFM from Line Features Given images of lines:

33. GENERALIZATION – Planar Features Homogeneous representation of a 3-D plane Projection of a planar point to the image Projection of a planar line to the image

34. GENERALIZATION – Multiple View Matrix: Coplanar Features Given that a point and line features lie on a plane in 3-D space: Besides multilinear constraints, it simultaneously gives homography:

35. GENERALIZATION – Coplanar Point/Line Duality On the plane any two points determine a line and any two lines determine a point. Theorem [Point/Line Duality] For planar features, points and lines are hence equivalent!

36. GENERALIZATION – SFM from Coplanar Features On the plane a set of points is equivalent to a set of lines, vice versa. • One can use either planar or to solve SFM as in the generic • point and line case. Algorithms need only minor changes. • Rank deficiency of planar or exploits multilinear constraints • and homography constraints simultaneously.

37. GENERALIZATION – 3-D Curves & Surfaces Differentiating the matrix of a point (moving) along a curve: gives rise to rank deficiency condition for curve. intensity level sets region boundaries . . .

38. GENERALIZATION – From Tangent to Point-wise Correspondence gives rise to a set of ordinary differential equations: • The rank deficiency condition for relates points and tangent • lines of a curve. • Solving these equations establishes point-wise correspondence for • image curves and in fact eventually for surface as well. • gives constraints on curvature and normals of image curves.

39. CONCLUSIONS AND ON-GOING WORK • Rank deficiency condition simplifies and unifies existing algebraic • results in multilinear constraints (no tensor and algebraic geometry). • Rank deficiency condition exhibits clear geometric interpretation. • Rank deficiency condition unifies the study of point, line, curve and • even surface in 3-D. • Rank deficiency condition naturally reveals point/point and • point/line duality. • Rank deficiency condition gives rise to uniform linear algorithms for • feature matching, motion recovery and new view synthesis. • The results no longer discriminate two, three, four or multiple views, • nor Euclidean, affine or projective camera models. • Consistent, optimal and robust reconstruction of motion/structure. • Numerical algorithms for curve, surface reconstruction from m views. • Multiple views of multiple rigid body motions.

40. Multiple View Geometry Unified Kun Huang, Rene Vidal and Jana Kosecka by Yi Ma CSL Technical Report, UILU-ENG #01-2208 (DC-200), 05/08/01 CSL Technical Report, UILU-ENG #01-2209 (DC-201), 05/08/01