Geometric Principles of Multiple Visual Sensors

1. UIUC Ph.D. Preliminary Exam Geometric Principles of Multiple Visual Sensors Kun Huang P.R.Kumar and Yi Ma Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC http://black.csl.uiuc.edu/~kunh

2. INTRODUCTION GEOMETRY FOR MULTIPLE IMAGES:Rank condition for points GENERALIZED MULTIPLE VIEW GEOMETRY GEOMETRY FOR SINGLE IMAGES:With knowledge of scene VISION-BASED GUIDANCE AND CONTROL FUTURE WORK

3. INTRODUCTION GEOMETRY FOR MULTIPLE IMAGES: Rank condition for points GENERALIZED MULTIPLE VIEW GEOMETRY GEOMETRY FOR SINGLE IMAGES: With knowledge of scene VISION-BASED GUIDANCE AND CONTROL FUTURE WORK

4. INTRODUCTION – The Fundamental Problem Input: Corresponding “features” in multiple images. Output: Camera motion, camera calibration, object structure. Jana’s apartment Ignoring other pictorial cues: texture, shading, contour, etc.

5. INTRODUCTION – History of “Modern” Geometric Vision • Chasles, formulated the two-view seven-point problem,1855 • Hesse, solved the above problem, 1863 • Kruppa, solved the two-view five-point problem, 1913 • Longuet-Higgins, the two-view eight-point algorithm, 1981 • Huang and Faugeras, SVD based eight-point algorithm, 1989 • Liu and Huang, the three-view trilinear constraints, 1986 • Tomasi & Kanade, orthographic factorization, 1992 • Han & Kanade, factorization for dynamical scenes, 2001 • Sturm & Triggs, projective factorization, 1995 • Triggs, the four-view quadrilinear constraints, 1995 • Hartley & Zisserman, static scene multiple view constraints • Wolf & Shashua, multiple view constraints for dynamical scenes 2001 • Ma, Huang et. al., the multiple-view rank condition, 2001 • Huang, Fossum and Ma, generalized rank conditions, 2002 • Hong, Huang, Yang and Ma, symmetric rank conditions, 2002-2003

6. Rank Conditions INTRODUCTION – The Fundamental Problem curve & surface plane algorithm line projective algebra point affine geometry Euclidean perspective orthographic 2 views omni-directional 3 views 4 views m views

7. INTRODUCTION – Are Multiple Views Necessary?

8. INTRODUCTION – A Little Notation The “hat” of a vector:

9. INTRODUCTION – Image of a Point Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane

10. INTRODUCTION GEOMETRY FOR MULTIPLE IMAGES:Rank condition for points GENERALIZED MULTIPLE VIEW GEOMETRY GEOMETRY FOR SINGLE IMAGES: With knowledge of scene VISION-BASED GUIDANCE AND CONTROL FUTURE WORK

11. GEOMETRY FOR MULITPLE IMAGES – Rank Constraints Two-view correspondence x1,x2 andT2 are coplanar

12. . . . GEOMETRY FOR MULITPLE IMAGES – Rank Constraints Corresponding point features

13. GEOMETRY FOR MULITPLE IMAGES – Rank Constraints Mp encodes exactly the 3-D information missing in one image.

14. For the jth point SVD Iteration For the ith image SVD GEOMETRY FOR MULITPLE IMAGES – Reconstruction Algorithms Given m images of n(>7) points

15. 90.840 89.820 GEOMETRY FOR MULITPLE IMAGES – Reconstruction Algorithms

16. GEOMETRY FOR MULITPLE IMAGES – Reconstruction Algorithms

17. INTRODUCTION GEOMETRY FOR MULTIPLE IMAGES: Rank condition for points GENERALIZED MULTIPLE VIEW GEOMETRY GEOMETRY FOR SINGLE IMAGES: With knowledge of scene VISION-BASED GUIDANCE AND CONTROL FUTURE WORK

18. Before: Now: Time base GENERALIZED MULTIPLE VIEW GEOMETRY – Euclidean Embedding For a fixed camera, assume the point moves with constant acceleration:

19. Inclusion Singlehyperplane Restriction to a hyperplane Intersection GENERALIZED MULTIPLE VIEW GEOMETRY– Incidence Relations

20. GENERALIZED MULTIPLE VIEW GEOMETRY – Incidence Relations “Pre-images” are all incident at the corresponding features. . . .

21. Its image corresponds to a (p+1)- dimensional subspace in : The orthogonal complementary of s is a (k-p)-dimensional subsapce in , it is denoted as coimage : GENERALIZED MULTIPLE VIEW GEOMETRY – Image and Coimage Image and coimage of a p-dimensional hyperplane in

22. We define the multiple view matrix as: where ‘s and ‘s are images and coimages of hyperplanes. GENERALIZED MULTIPLE VIEW GEOMETRY – Generalized Matrix Projection from n-dimensional space to k-dimensional space:

23. GENERALIZED MULTIPLE VIEW GEOMETRY – Generalized Matrix Projection from n-dimensional space to k-dimensional space:

24. Projection from to Projection from to GENERALIZED MULTIPLE VIEW GEOMETRY – Rank Conditions Singlehyperplane

25. Projection from to Projection from to GENERALIZED MULTIPLE VIEW GEOMETRY – Rank Conditions Inclusion

26. Projection from to Projection from to GENERALIZED MULTIPLE VIEW GEOMETRY – Rank Conditions Intersection

27. Projection from to Restriction Appending one block to , all the above results hold. Projection from to Appending one block to , all the above results hold. GENERALIZED MULTIPLE VIEW GEOMETRY – Rank Conditions

28. A single rank condition expresses the incidence condition among the vertex and three edges in terms of their three images. GENERALIZED MULTIPLE VIEW GEOMETRY – Rank Conditions

29. GENERALIZED MULTIPLE VIEW GEOMETRY – Landing of UAV Rate: 10Hz Accuracy: 5cm, 4o Berkeley Aerial Robot (BEAR) Project

30. INTRODUCTION GEOMETRY FOR MULTIPLE IMAGES: Rank condition for points GENERALIZED MULTIPLE VIEW GEOMETRY GEOMETRY FOR SINGLE IMAGES:With knowledge of scene VISION-BASED GUIDANCE AND CONTROL FUTURE WORK

31. Perception from Single View Structure Hypothesis Testing 3D Locations Multiple View Geometry Feature Correspondences GEOMETRY FOR SINGLE IMAGE – Visual Perception Bottom-Up Approach Multiple Images

32. Structure & 3D Locations Multiple View Geometry Hypothesis Testing Features GEOMETRY FOR SINGLE IMAGE – Visual Perception Hypothesis-Testing Approach Image

33. GEOMETRY FOR SINGLE IMAGE – Symmetry

34. GEOMETRY FOR SINGLE IMAGE – Symmetry Symmetry captures almost all “regularities”.

35. Symmetry on object (1) 2 (2) 1 (3) 4 (4) 3 Virtual camera-camera GEOMETRY FOR SINGLE IMAGE – Hidden Images in Each View Symmetry-based reconstruction = hidden image + rank condition

36. 2 pairs of symmetric points 2(1) 1(2) Reflective homography 4(3) 3(4) Decompose H to obtain (R’, T’, N) and T0 Solve Lyapunov equation to obtain R0. GEOMETRY FOR SINGLE IMAGE – Reflective Homography

37. GEOMETRY FOR MULTIPLE IMAGE – With Scene Knowledge • 3-D reconstruction from multiple views with symmetry is • simple, accurate and robust!

38. ? GEOMETRY FOR MULTIPLE IMAGE – Alignment of Different Objects

39. GEOMETRY FOR MULTIPLE IMAGE – Scale Correction For a point p on the intersection line

40. GEOMETRY FOR MULTIPLE IMAGE – Alignment of Different Views

41. For any image x1 in the first view, its corresponding image in the second view is: GEOMETRY FOR MULTIPLE IMAGE – Scale Correction

42. GEOMETRY FOR MULTIPLE IMAGE – Alignment of Multiple Views Method is object-centered and baseline-independent.

43. GEOMETRY FOR MULTIPLE IMAGE – Experimental Results

44. GEOMETRY FOR MULTIPLE IMAGE – Experimental Results

45. GEOMETRY FOR MULTIPLE IMAGE – Image Transfer 1

46. GEOMETRY FOR MULTIPLE IMAGE – Camera Poses

47. GEOMETRY FOR MULTIPLE IMAGE – Full 3-D Model

48. GEOMETRY FOR MULTIPLE IMAGE – CSL Building

49. GEOMETRY FOR MULTIPLE IMAGE – Curved Surface Inside the foyer of Beckman Institute

50. GEOMETRY FOR MULTIPLE IMAGE – Curved Surface Inside the foyer of Beckman Institute