Rank Conditions in Multiple View Geometry: A Comprehensive Study

1. ACCV02, Melbourne Rank Conditions in Multiple View Geometry Kun Huang (ECE.UIUC), Jana Kosecka (CS.GMU) by Yi Ma Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC http://black.csl.uiuc.edu/~yima

2. FORMULATION: camera model and multiple images MAIN RESULT: rank conditions v.s. multifocal constraints GENERALIZATION: coplanar features… APPLICATIONS: structure from motion CONCLUSIONS AND ON-GOING WORK

3. FORMULATION - Fundamental Geometric Problem Input: Corresponding images (of “features”) in multiple images. Output: Camera motion, camera calibration, object structure. Jana’s apartment

4. FORMULATION – Literature Review “Multiple” view geometry theory • Two views: Kruppa’13, Longuet-Higgins’81, Huang & Faugeras’89, … • Three views: Liu & Huang’86, Spetsakis & Aloimonos’90, Shashua’94, … • Four views: Triggs’95, Shashua’00, … • Multiple views: Heyden & Astrom’97’98, … • Higher dimension: Wolf & Shashua’01 Multiple view geometry algorithms • Euclidean: Maybank’93, Weng, Ahuja & Huang’93, … • Affine: Quan & Kanade’96, … • Projective: Triggs & Sturms’96, … • Orthographic: Tomasi & Kanade’92, … Recent books on multiple view geometry 1. Multiple view geometry in computer vision, Hartley & Zisserman’00. 2. Geometry of multiple images, Faugeras, Luong & Papadopoulo’01.

5. Rank Conditions FORMULATION – A Provocative Stand: One Theorem for All & More? curve & surface plane line algorithm projective point algebra affine geometry Euclidean perspective orthographic 2 views omni-directional 3 views 4 views m views

6. FORMULATION – A Little Notation: Hat Operator

7. FORMULATION - Pinhole Camera Model Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane

8. FORMULATION – Image of a Line Feature Homogeneous representation of a 3-D line Homogeneous representation of its 2-D co-image Projection of a 3-D line to an image plane

9. MAIN RESULT – Incidence Relations Among Features Multiview constraints are nothing but incidence relations at play! . . .

10. Multi-nonlinear constraints • among 3, 4-wise images. • Multi-linear constraints • among 2, 3-wise images. MAIN RESULT – What is the (Multiple View) Matrix? Theorem [The Universal Rank Condition] for images of a point on a line:

11. MAIN RESULT – Some Instances with Mixed Features Examples: Case 1: a line reference Case 2: a point reference • All previously known constraints are the theorem’s instances. • Degenerate configurations if and only if a drop of rank.

12. MAIN RESULT – Compared to Conventional Multifocal Constraints For images of the same 3-D point : (leading to the conventional approach) Multilinear constraints among 2, 3, 4-wise views

13. MAIN RESULT – Rank Condition For the Point Feature WLOG, choose camera frame 1 as the reference Multiple View Matrix Lemma [Rank Condition for Point Features] Let then and are linearly dependent.

14. MAIN RESULT – Rank Condition Implies Multifocal Constraints • These constraints are only necessary but NOT sufficient! • However, there is NO further relationship among quadruple wise • views. Quadrilinear constraints hence are redundant!

15. MAIN RESULT – Multiple View Matrix: Line v.s. Point Point Features Line Features

16. MAIN RESULT – Rank Conditions: Line v.s. Point

17. MAIN RESULT – Global Multiple-View Analysis (Example)

18. MAIN RESULT – A Family of Intersecting Lines each can randomly take the image of any of the lines: Nonlinear constraints among up to four views . . .

19. GENERALIZATION – Restriction to a Plane (A Corollary) Homogeneous representation of a 3-D plane Corollary [Coplanar Features] Rank conditions on the new extended remain exactly the same!

20. In addition to previous constraints, it simultaneously gives homography: GENERALIZATION – Multiple View Matrix: Coplanar Features Given that a point and line features lie on a plane in 3-D space:

21. Multi-quadratic equations in given images 4 coplanar points + 3 virtual points = 7 effective points GENERALIZATION – Intrinsic Rank Condition for Planar Features

22. APPLICATIONS – Motion and Structure from Multiple Views Given images of points:

23. APPLICATIONS – SVD Based Four Step Algorithm for SFM

24. APPLICATIONS – Utilize All Incidence Conditions (Example) Three edges intersect at each vertex. . . .

25. APPLICATIONS – Utilizing All Incidence Conditions (Simulations)

26. APPLICATIONS – Landing a Helicopter (Experiments) Images courtesy of Omid Shakernia, UCB

27. CONCLUSIONS AND ON-GOING WORK • Rank condition simplifies, unifies and completes existing algebraic • results on multiview constraints (no tensor and algebraic geometry). • Rank condition is for all features, all incidence relations, all number • of views, all types of projection, all (linear) spaces of arbitrary • dimensions. • Rank condition intrinsically ties together geometry and algebra. • Metric multiple view geometry. • Consistent, optimal and robust algorithms for correspondence, • image-based view synthesis, and reconstruction of motion & structure. • Apply to dynamical scenes: human motion; sensor networks… • Real-time algorithms for autonomous navigation and robotic control.

28. Rank Conditions in Multiple View Geometry Kun Huang (ECE.UIUC), Jana Kosecka (CS.GMU) by yima@uiuc.edu “An invitation to 3-D vision”, Ma, Soatto, Kosecka, and Sastry, Springer-Verlag, expected 2002.