Multiple-view Reconstruction from Points and Lines

1. Multiple-view Reconstruction from Points and Lines

2. Calibrated case Recover Essential Matrix decompose to R,T 3D reconstruction Planar case Recover homography H decompose to R,T,n,d Uncalibrated case Recover Fundamental Matrix F Projective reconstruction Recover planar homography (no decomposition possible) Rotational homography (mosaics) Calibration with planar rig Calibration with 3D rig Single view Homography between 3D plane and image plane (rectification) Partial calibration using vanishing points Partial calibration and pose recovery using from single view/world homography Single and Two view summary

3. Problem formulation Input: Corresponding images (of “features”) in multiple images. Output: Camera motion, camera calibration, object structure.

4. Affine/Orthographic projection model • Full perspective projection model • Affine camera projection model (in-homogenous coordinates) • Good approximation if the distance from the scene >> scene depth variation • Rigid body motion under affine projection model

5. Affine Multiview Factorization • Problem – given n correspondences in m views determine and 3D points • Obtained by minimization of the following objective function • Assume that the centroid of the structure is the origin of the coordinate frame and denote the centroids in the following way • Minimize with respect to

6. Affine Multiview Factorization The choice of the frame is arbitrary – further assume that The objective function then becomes Writing all the constrains in the matrix form Drop ~ for clarity

7. Affine Multiview Factorization Measurement matrix Motion matrix Structure matrix • Matrix W must have rank 3 (product of two rank 3 matrices) • In noise case we seek best rank 3 approximation of W matrix • Given the actual measurement matrix compute SVD • Best rank 3 approximation is

8. Affine Multiview Factorization • Decomposition of is not unique • Affine ambiguity Q • The ambiguity can be resolved using rotation matrix constraints where [Tomasi, Kanade’IJCV 1992] Factorization approach

9. Projective Multiview Factorization Measurement matrix Motion matrix Structure matrix • Similar strategy – expect now the scales are unknown • We don’t have the matrix W • Matrix W is a product of two rank 4 matrices – i.e. is rank 4 • How to apply the factorization idea to projective setting ? • - need to compute scales (possible from two view reconstruction) • - or initialize the scales and iterate

10. Projective Factorization Algorithm • Given a set of image points in n views • Compute the projective depths using two view methods or set • Form the measurement matrix W and find the nearest rank 4 approximation using SVD • Decompose into camera matrices and structure • Optionally reproject the and iterate (i.e. given new motions, we can compute new scales)

11. Multi-view methods • Advantages of multiview methods - more frames – wider baseline – better conditioned algorithms - additional complexity of matching (establishing correspondences) across multiple views • How are multi-view and two view methods related ? • Can we just run the two view algorithm for each pair of views and get better results ? • What if we are trying to do reconstruction using lines ? • Are there other constraints between multiple views then pairwise epipolar constraints ? • Next – brief tour of multilinear constraints

12. Traditional multifocal constraints For images of the same 3-D point : (leading to the conventional approach) Multilinear constraints among 2, 3, 4-wise views

13. Rank conditions for point feature WLOG, choose camera frame 1 as the reference Multiple-View Matrix Lemma [Rank Condition for Point Features] Let then and are linearly dependent.

14. Rank conditions vs. multifocal constraints

15. Rank conditions vs. multi-focal constraints

16. Rank conditions vs. multifocal constraints • These constraints are only necessary but NOT sufficient! • However, there is NO further relationship among quadruple wise • views. Quadrilinear constraints hence are redundant!

17. Point Features – Uniqueness of the pre-image bilinear constraints – coplanarity constraints Extend to three views collinear optical centers coplanar optical centers

18. Point Features – Uniqueness of the pre-image trilinear constraints Given m vectors with respect to m camera frames, They correspond to a unique point in the 3D space if the rank of the Matrix Mp is 1. If rank is 0, the point is determined up to a line on which all optical centers must lie .

19. Image of a line feature Homogeneous representation of a 3-D line Homogeneous representation of its 2-D co-image Projection of a 3-D line to an image plane

20. Multiple-view matrix: line vs. point Point Features Line Features

21. Rank conditions: line vs. point

22. Multiple-view structure and motion recovery Given images of points:

23. SVD based 4-step algorithm for SFM

24. Utilizing all incidence relations Three edges intersect at each vertex. . . .

25. Example: simulations

26. Example: simulations

27. Example: experiments Errors in all right angles < 1o

28. Summary • Incidence relations <=> rank conditions • Rank conditions => multiple-view factorization • Rank conditions implies all multi-focal constraints • Rank conditions for points, lines, planes, and • (symmetric) structures.

29. Global multiple-view analysis: examples

30. A family of intersecting lines each can randomly take the image of any of the lines: Nonlinear constraints among up to four views . . .

31. Universal rank condition • Multi-nonlinear constraints • among 3, 4-wise images. • Multi-linear constraints • among 2, 3-wise images. Theorem [The Universal Rank Condition] for images of a point on a line:

32. Instances with mixed features Examples: Case 1: a line reference Case 2: a point reference • All previously known constraints are the theorem’s instances. • Degenerate configurations if and only if a drop of rank.

33. Generalization – restriction to a plane Homogeneous representation of a 3-D plane Corollary [Coplanar Features] Rank conditions on the new extended remain exactly the same!

34. In addition to previous constraints, it simultaneously gives homography: Generalization – restriction to a plane Giventhat a point and line features lie on a plane in 3-D space: GENERALIZATION – Multiple View Matrix: Coplanar Features