3D Reconstruction from Images

1. 3D Reconstruction from Images 2009, Siggraph A complete pipeline from pixels to finalized 3D objects through object segmentation and recognition.

2. Vision geometry • Structure from motion, or 3D reconstruction, or SLAM • Example par excellence of modern computer vision integrating feature detection, object recognition and geometry computation • Camera poses • Positioning, visual GPS, for GPS denied environments such as indoors, cities … • Localization, for automation by machine vision • 3D reconstruction • Obstacle avoidance, navigation, planning, and environment learning

3. The objects of study in geometry • Lines to points • Corners to feature points • Match correlation to descriptors of high dimension • Descriptors to recognition, beyond the geometry scope • More deterministic computational models • Robust statistics • Large-scale

4. New book Image-based Modeling, Long QUAN, Springer-Verlag, 2010.

5. My perspective Part I What is computer vision? What is 3D reconstruction? From pixels to 3D points Structure from motion A quasi-dense approach From 3D points to objects Small-scale objects Smooth surfaces, Hairs, Trees Large-scale buildings Façade, Buildings, Cities Part II Large-scale automatic 3D mapping Conclusions

6. Overview • Introduction to projective geometry • 1 view geometry (calibration, …) • 2-view geometry (stereo, motion, …) • 3- and N-view geometry • Autocalibration (metric reconst.) • Application

7. Basic geometric concepts to understand • Affine, Euclidean geometries (inhomogeneous coordinates) • projective geometry (homogeneous coordinates) • plane at infinity: affine geometry • absolute conic: Euclidean geometry

8. Introduction to projective geometry • Intuitive ideas from projective geometry • (Formal definition of projective spaces)

9. Intuitive introduction Naturally everything starts from the known vector space • add two vectors • multiply any vector by any scalar • zero vector – origin • finite basis

10. Vector space to affine: isomorph, one-to-one • vector to Euclidean as an enrichment: scalar prod. • affine to projective as an extension: add ideal elements Pts, lines, parallelism Angle, distances, circles Pts at infinity

11. Points at infinity: Algebraic extension to pts at infinity: introduction of homogeneous coordiantes Rq: the homogeneous coordinates are not unique, up to a scale.

12. On a plane, Can we see the pts at infinity? The direction d is a pt at infinity:

13. Provisional summary a projective space is an affine space + some pts at infinity or a projective space is a space of ‘homogeneous coordinates’

14. ((Formal) definition of projective geometry) Given K=R or C, can be defined as the nonzero equivalent classes determined by the relation ~ on If there is non-zero real number such that Any element of the equivalent class will be called the homogeneous coordinates of the point.

15. A projective space is nothing but a quotient space (space of equivalent classes): • A space of homogeneous coordinates • Basic structure: linear dependence of points Definition: a pt x is said to be linearly dependent on a set of pts if

16. Relation between Pn (homo) and Rn (in-homo): Rn --> Pn, extension, embedded in Pn --> Rn, restriction, P2 and R2

17. One example of construction of projective line by quotient space

18. Examples of projective spaces • Projective plane P2 • Projective line P1 • Projective space P3

19. Projective plane Space of homogeneous coordinates (x,y,t) Pts are elements of P2 Pts are elements of P2 Pts at infinity: (x,y,0), the line at infinity 4 pts determine a projective basis 3 ref. Pts + 1 unit pt to fix the scales for ref. pts Relation with R2, (x,y,0), line at inf., (0,0,0) is not a pt

20. Lines: Linear combination of two algebraically independent pts Operator + is ‘span’ or ‘join’ Line equation:

21. Point coordinate, column vector A line is a set of linearly dependent points Two points define a line Line coordinate, row vector A point is a set of linearly dependent lines Two lines define a point Point/line duality: • What is the line equation of two given points? • ‘line’ (a,b,c) has been always ‘homogeneous’ since high school!

22. Given 2 points x1 and x2 (in homogeneous coordinates), the line connecting x1 and x2 is given by Given 2 lines l1 and l2, the intersection point x is given by NB: ‘cross-product’ is purely a notational device here.

23. Compute the intersection point of two lines, each defined by two points

24. Conics Conics: a curve described by a second-degree equation • 3*3 symmetric matrix • 5 d.o.f • 5 pts determine a conic • affine classification with pts at inf • the line tangent to a conic at a pt • dual conic • pole and polar • one numerical example

25. Tangent to a conic at a pt x on C is given by l=Cx Dual conic (in line coordinates) is given by l^T C^{-1} l = 0 Polar of a pt x is l = C x and (is also a tangent on C from x if x is on C) Conjugacy: a pt y on l, y^T l = 0, y^T C x = 0 (in Eucl. Ortho: y^T x = 0)

26. Projective classification of (point) conics: General rank 3: x^2+y^2+t^2=0 (imaginary) x^2+y^2-t^2=0 Degenerate conics

27. Affine classification: Line at infinity

28. Projective line Homogeneous pair (x1,x2) Finite pts: Infinite pts: how many? Topology? A basis by 3 pts Fundamental inv: cross-ratio

29. Euclidean coordinate: the distance • Affine coordinate: the ratio of the distances (x-a/a-o) • Projective coordinate: the ratio of the ratio of the distances (cross-ratio, double ratio) ((x-a)/(a-o)) / ((x-b)(b-o))

30. Projective space P3 • Pts, elements of P3 • Relation with R3, plane at inf. • lines: linear comb of 2 pts, but 3*4 matrix, complicated …back later • planes: linear comb of 3 pts • Basis by 4 (ref pts) +1 pts (unit) • quadrics: two classes---ruled and unruled • (topology of P3) Line equation? Plane equation: ...

31. planes In practice, take SVD Homework: compute plane normal vector?

32. Plucker coordinates of lines in P3 How many d.o.f??? 6 2*2 minors, Two lines intersect in space iff

33. (Quadric surfaces) Ruled: hyperboloid of one sheet, 1,1,-1,-1---topo torus Unruled: sphere, ellipsoid, hyperboloid and paraboloid: 1,1,1,-1 ---- topo sphere

34. Key points • Homo. Coordinates are not unique • 0 represents no projective pt • finite points embedded in proj. Space (relation between R and P) • pts at inf. (x,0) missing pts, directions • hyper-plane (co-dim 1): • duality between u and x,

35. Introduction to transformation 2D general Euclidean transformation: 2D general affine transformation: 2D general projective transformation:

36. Projective transformation = collineation = homography Consider all functions All linear transformations are represented by matrices A Note: linear but in homogeneous coordinates!

37. Properties • (n+1)*(n+1) -1 d.o.f. • all projective properties are left invariant by A • all transformations form a group GL(n,R) N+2 pts to determine a trans. = a proj. basis Check the most important one: linear dependency, i.e. lines into lines as line is just a span Starting pt for new investigation: Klein’s Erlangen program Inversely, we may also prove that any 1-1 transf. Preserving lines is a linear trans in homogeneous coord.

38. (Some examples of transformations) on pts, lines and conics: Transforms contravariantly Co-variantly to preserve incidence Co-variantly NB: co-,contra-variance is w.r.t. the basis trans. Transpose is of no importance, il accommodates row/column vectors Some numerical examples of transformation on P2

39. How to compute (canonical or standard) coordinates?--- affine case Given 4 pts, x1, x2, x3, x4, find the affine coord of x4 w.r.t. x1, x2 and x3:

40. How to compute (canonical or standard) coordinates?--- affine case Given 4 pts, x1, x2, x3, x4, find the affine coord of x4 w.r.t. x1, x2 and x3: • by definition, vector(x4-x1) = a vector(x2-x1) + b vector(x3-x1) • by canonical transformation, x1->(0,0), x2->(1,0), x3->(0,1), get transfromation A, then Ax4 How to solve Ax=b?

41. Canonical projective coordinates? Given 5 pts, x1, x2, x3, x4, x5find the affine coord of x5 w.r.t. x1, x2, x3, x4: By canonical transformation: How to solve Ax=0?

42. A transformation between 2 spaces?

43. Exercise Compute the transformation from (0,0,1), (1,0,1), (0,1,1) and (1,1,1) into (0,0,1), (1,1/4,1),(0,1,1) and (1,3/4,1)

44. (Geometry as an invariant theory of transformation groups) • projective geom. GL(n,R) cross-ratio • affine geom. Subgroup A(n,R) ratio • Euclidean geom. Subgroup E(n,R) distance All proj. Transformations nicely form a group! Each geometry is associated with a (sub)group! Hierarchy of geometry:

45. Example of dim 2 From projective to affine: Affine transformation is a projective one which leaves the line at inf. invariant: x3=x3’=0

46. From affine to euclidean Similarity transformation is an affine one which leaves the circular pts I and J invariant What are the circular points?

47. Circular points The line at infinity of a usual plane The pair of circular points Intuitive introduction of circular pts

48. Example of transformation in P3 Affine transformation leaves the plane at inf. invariant Similarity (euclidean) leaves the absolute conic (globally, not point-wise) invariant What is the absolute conic?

49. A space conic on the plane at infinity: In point coordinates: In plane coordinates: rank 3 space quadric=absolute quadric (Absolute conic) Euclidean structure in projective space by the absolute conic

50. The absolute conic The plane at infinity The line at infinity of a usual plane The pair of circular points A usual plane in 3D