by Yi Ma

1. Rank Conditions in Multiple View Geometry Rene Vidal (EECS.UCB), Kun Huang (ECE.UIUC) Jana Kosecka (CS.GMU), Robert Fossum (Math.UIUC) by Yi Ma Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC http://black.csl.uiuc.edu/~yima

2. FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v.s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

3. FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v.s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

4. FORMULATION - Fundamental Geometric Problem Input: Corresponding images (of “features”) in multiple images. Output: Camera motion, camera calibration, object structure. Jana’s apartment Image courtesy of Jana Kosecka

5. curve & surface plane line algorithm projective point algebra affine geometry Euclidean perspective orthographic 2 views omni-directional 3 views 4 views m views FORMULATION – Orthodox View (State of the Art)

6. FORMULATION – Literature Review “Multiple” view geometry theory • Two views: Kruppa’13, Longuet-Higgins’81, Huang & Faugeras’89, … • Three views: Spetsakis & Aloimonos’90, Shashua’94, Hartley’94, … • Four views: Triggs’95, Shashua’00, … • Multiple views: Heyden & Astrom’97’98, … • Higher dimension: Wolf & Shashua’01 Multiple view geometry algorithms • Euclidean: Maybank’93, Weng, Ahuja & Huang’93, … • Affine: Quan & Kanade’96, … • Projective: Triggs’96, … • Orthographic: Tomasi & Kanade’92, … Recent books on multiple view geometry 1. Multiple view geometry in computer vision, Hartley & Zisserman’00. 2. Geometry of multiple images, Faugeras, Luong & Papadopoulo’01.

7. Rank Conditions FORMULATION – A Provocative Stand: One Theorem for All & More? curve & surface plane line algorithm projective point algebra affine geometry Euclidean perspective orthographic 2 views omni-directional 3 views 4 views m views

8. FORMULATION – A Little Notation: Hat Operator

9. FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v.s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

10. POINT FEATURE – Pinhole Camera Model Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane

11. POINT FEATURE – Multiple View Structure From Motion Given corresponding images of points: recover everything else from equations associated to each: . . .

12. POINT FEATURE – Conventional Multilinear (Multifocal) Constraints For images of the same 3-D point : (leading to the conventional approach) Multilinear constraints among 2, 3, 4-wise views

13. POINT FEATURE – Rank Condition on the Multiple View Matrix WLOG, choose camera frame 1 as the reference Multiple View Matrix Lemma [Rank Condition for Point Features] (generic) (degenerate) Let then and are linearly dependent.

14. POINT FEATURE – Rank Condition Implies Bilinear Constraints Fact: Given non-zero vectors Hence, we have These constraints are only necessary but NOT sufficient!

15. POINT FEATURE – Rank Condition Implies Trilinear Constraints Fact: Given non-zero vectors Hence, we have • These constraints are only necessary but NOT sufficient! • However, there is NO further relationship among quadruple wise • views. Quadrilinear constraints hence are redundant!

16. POINT FEATURE – Uniqueness of Pre-image (Bilinear Constraints) “Bilinear means pair-wise coplanar”: except in a rare coplanar case: Rectilinear motion Trifocal plane

17. POINT FEATURE – Uniqueness of Pre-image (Trilinear Constraints) “Trilinear means triple-wise incidental”: except in a rare collinear case:

18. POINT FEATURE – Uniqueness of Pre-image (Multiple View Matrix) Proposition [Uniqueness of Pre-image] Given vectors with respect to camera frames, they correspond to a unique point in the 3-D space if the rank of the matrix is1. If the rank is 0, the point is determined up to a line on which all the camera centers must lie. “point incidence condition” . . .

19. POINT FEATURE – Geometric Interpretation is the “depth” of the point relative to the camera center. Points that give the same matrix are on a sphere of radius

20. FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v.s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

21. GENERALIZATION – Line Feature Homogeneous representation of a 3-D line Homogeneous representation of its 2-D co-image Projection of a 3-D line to an image plane

22. GENERALIZATION – Multiple View Matrix: Line v.s. Point Point Features Line Features

23. GENERALIZATION – Rank Conditions: Line v.s. Point

24. GENERALIZATION – Incidence Relations Among Features Incidence conditions: inclusion, intersection, and restriction. “nothing but incidence conditions” . . .

25. Multi-nonlinear constraints • among 3, 4-wise images. • Multi-linear constraints • among 2, 3-wise images. GENERALIZATION – What is the Matrix? Theorem 1 [The Universal Rank Condition] for images of a point on a line:

26. Examples: Case 1: a line reference Case 2: a point reference GENERALIZATION – Implications of the Rank Condition • All previously known constraints are the theorem’s instantiations. • It implies more constraints and is now complete. • Degenerate configurations if and only if a drop of rank.

27. GENERALIZATION – Global Multiple-View Analysis (Example)

28. GENERALIZATION – A Family of Incidental Lines (A Corollary) each can randomly take the image of any of the lines: Nonlinear constraints among up to four views . . .

29. Corollary [Coplanar Features] Rank conditions on the new extended remain exactly the same! GENERALIZATION – Restriction to a Plane (A Corollary) Homogeneous representation of a 3-D plane

30. In addition to previous constraints, it simultaneously gives homography: GENERALIZATION – Multiple View Matrix: Coplanar Features Given that a point and line features lie on a plane in 3-D space:

31. Corollary [Point/Line Duality] For planar features, points and lines are hence equivalent! GENERALIZATION – Coplanar Point/Line Duality On the plane any two points determine a line and any two lines determine a point.

32. Multi-quadratic equations in given images GENERALIZATION – Intrinsic Rank Condition for Planar Features 4 coplanar points + 3 virtual points = 7 effective points

33. Before: Now: Time base This is a perspective projection from <n to <2. GENERALIZATION – Euclidean Imbedding of Dynamic Scenes

34. Theorem 2 [Generalized Rank Condition for from <n to <k] GENERALIZATION – Rank Condition in Space of High Dimension

35. GENERALIZATION – Rank Conditions for Curves & Surfaces Differentiating the matrix of a point (moving) along a curve: gives rise to a rank condition for curve. intensity level sets region boundaries . . .

36. GENERALIZATION – From Tangent to Point-wise Correspondence gives rise to a set of ordinary differential equations: • The rank deficiency condition for relates points and tangent • lines of a curve. • Solving these equations establishes point-wise correspondence for • image curves and in fact eventually for surface as well. • gives constraints on curvature and normals of image curves.

37. FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v.s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

38. APPLICATIONS – Multiple View Matching Test Given the projection matrix associated to camera frames. Then for vectors

39. APPLICATIONS – Transferring Images to a New View Given the projection matrix associated to camera frames. Then for given vectors So given images, rank deficiency adds a linear constraint on the image. Computing the kernel of gives the new image.

40. APPLICATIONS – Motion and Structure from Multiple Views Given images of points:

41. APPLICATIONS – SVD Based Four Step Algorithm for SFM

42. APPLICATIONS – Utilize All Incidence Conditions (Example) Three edges intersect at each vertex. . . .

43. APPLICATIONS – Utilizing All Incidence Conditions (Simulations)

44. APPLICATIONS – Landing a Helicopter (Experiments) Images courtesy of Omid Shakernia, UCB

45. FORMULATION: camera model and multiple images POINT FEATURE: multilinear constraints v.s. rank conditions GENERALIZATION: line, plane, space of higher dimensions APPLICATIONS: matching, transfer, structure from motion CONCLUSIONS AND ON-GOING WORK

46. CONCLUSIONS AND ON-GOING WORK • Rank condition simplifies, unifies and completes existing algebraic • results on multiview constraints (no tensor and algebraic geometry). • Rank condition is for all features, all incidence relations, all number • of views, all types of projection, all (linear) spaces of arbitrary • dimensions. • Rank condition intrinsically ties together geometry and algebra. • Metric multiple view geometry. • Consistent, optimal and robust algorithms for correspondence, • image-based view synthesis, and reconstruction of motion & structure. • Apply to dynamical scenes: human motion; sensor networks… • Real-time algorithms for autonomous navigation and robotic control.

47. Rank Conditions in Multiple View Geometry Rene Vidal (EECS.UCB), Kun Huang (ECE.UIUC) Jana Kosecka (CS.GMU), Robert Fossum (Math.UIUC) by yima@uiuc.edu “Rank conditions on the multiple view matrix”, submitted to IJCV. “General rank conditions in multiple view geometry”, submitted to D&CG “An invitation to 3-D vision”, Ma, Soatto, Kosecka, and Sastry, 2002.