Day 2: Structure from motion diagram

1. Day 2: Structure from motion diagram Projective geometry Camera geometry Two-view geometry Three-view geometry n-view geometry Calibration

2. First topic: projective geometry Projective geometry HZ 2,3 Camera geometry Two-view geometry Three-view geometry n-view geometry Calibration

3. But a short introduction to the camera first Projective geometry HZ 2,3 Camera geometry Two-view geometry Three-view geometry n-view geometry Calibration

4. Introduction to the camera • Faugeras: the camera as ‘projective engine’: “A camera is a particular geometric engine which constructs planar images of the 3-dimensional world through a projection.” [FL 174] • we want to invert this imaging process to yield a model of the 3-dimensional world • must understand image formation

5. Theory of perspective in history • pinhole cameras and the image formation process were first explored by Renaissance painters, where it was called the theory of perspective • Leonardo da Vinci (quoting John Pecham): “Perspective ... should be preferred above all man’s discourses and disciplines. In this subject the visual rays are elucidated by means of demonstrations which derive their glory not only from mathematics but also from physics; the one is adorned equally with the flowers of the other.” [Kemp, p. 5] • Brunelleschi, a Florentine architect (Baptistery, Duomo) developed theory of perspective circa 1413 • theory  perspective machines  camera obscura  daguerrotype  camera • Faugeras cover + FL176 + camera obscura FL177 • need a dark room to capture just the light from the pinhole • used by painters as a portable device (see Vermeer in The Girl with a Pearl Earring) • lenses added at pinhole to gather light better • Daguerre replaced one wall of camera obscura with photosensitive film, yielding first photograph (daguerrotype) • et voila, the camera

6. Camera obscura

7. Theory of perspective 2 • behaviour at infinity immediately became of interest, as vanishing points were incorporated into paintings • vanishing point in Bosse: FL205 • vanishing line in Houckgeest: FL206 • renewed interest with the invention of cameras, leading to photogrammetry (theory of measurement of 3d world from 2d images): FL174 + Pollefeys CACM02 • projective geometry developed in mathematics: a tool that can be used to understand the image formation process

8. Pinhole camera model • geometrically very simple, encoding perspective projection • camera center = optical center C • image plane = retinal plane P • image x of X = intersection of XC with P • practical setup: • two screens: image plane and focal plane • pinhole in focal plane

9. Camera terminology • camera center, image plane • confusion: HZ (153) equate the image and focal planes; but for Faugeras (F35), the focal plane passes through the optical center • focal length f = dist (C, P) • focal plane = plane through C parallel to P • principal axis = line through C normal to P = optical axis • principal point = intersection of principal axis and P

10. Perspective projection • not a linear map in Euclidean geometry! • suppose • camera center = origin • principal axis = z-axis • (X,Y,Z)  f/Z (X,Y) in image plane Z=f • this is not linear

11. Projective geometry simplifies • perspective projection will be a linear map in projective geometry • what are other motivations for using projective geometry rather than Euclidean? • projective geometry is a simpler generalization of Euclidean geometry • lifting up a dimension simplifies, just as lifting from flatland to 3-space simplifies interpretation of flatland (e.g., sphere passing through flatland) • so (a) less special cases • e.g., translation vs. rotation/scaling (b) algebra is often simplified (homogeneous coordinates) • can reason about geometry at infinity • infinity is one of those special cases for Euclidean geometry • points and lines are treated equivalently (dual), so they interact well

12. First topic: projective geometry Projective geometry HZ 2,3 Camera geometry Two-view geometry Three-view geometry n-view geometry Calibration

13. Projective geometry diagram(2-space) Primitives pt/line/conic HZ 2.2 Behaviour at infinity Projective transform HZ 2.3 Hierarchy of maps Invariants HZ 2.4 DLT alg HZ 4.1 Rectification HZ 2.7

14. First topic: primitives Primitives pt/line/conic HZ 2.2 Behaviour at infinity Projective transform HZ 2.3 Hierarchy of maps Invariants HZ 2.4 DLT alg HZ 4.1 Rectification HZ 2.7

15. Point/line diagram point x y w point on line P . L = 0 line (a,b,c) line/line intersection L1 x L2 L1 L2 intersection P1 P2 point/point join P1 x P2 join

16. Point-line representation • we want a new data structure (or encoding) for points and lines • importance of representation and notation: think of pre-20th century mathematics notation (e.g., Descartes) • consider the implicit representation of a line in 2-space ax+by+c=0 • the line’s homogeneous representation is (a,b,c) • coefficients of the line’s implicit equation • for consistency with tensors later, let us treat line as a row vector (although HZ introduce as a column); OK to treat lines and points as generic vectors at early stages • consider a point in 2-space (x,y) • its homogeneous representation is (x,y,1) • coordinates in the implicit equation • a column vector • what are the ramifications of this encoding? • point/line incidence, intersection and join are all simplified • point P lies on line L iff P.L = 0 • intersection of two lines L1 and L2 is L1 x L2 (recall cross product) • join of two points P1 and P2 is P1 x P2 • HZ 26-27

17. Proofs • the point P lies on the line L iff P.L = 0 • after all, point satisfies line equation • dually, L intersects P iff L.P = 0 • recall that inner product = 0 encodes orthogonality • intersection of two (distinct) lines L1 and L2 is L1 x L2 • think geometrically: L1 x L2 is normal to L1 and L2 • so (L1xL2) . L1 = 0 and (L1xL2) . L2 = 0 • so the point L1xL2 lies on L1 and L2 • e.g., what is the intersection of x=1 and y=1? • join of two (distinct) points P1 and P2 is P1xP2 • again (P1xP2) . P1 = 0 and (P1xP2) . P2 = 0, so this line contains these two points • morals • inner product encodes incidence • this is also simple matrix multiplication if we represent points as columns and lines as rows • cross product encodes intersection and join

18. Exercise • think of inputting lines and computing their intersections using these projective results • points  lines  intersections

19. Homogeneous coordinates • the point and line encodings are vectors in projective space, with homogeneous representations • note that ax+by+c=0 is equivalent to kax + kby + kc = 0 • (a,b,c) = k(a,b,c) for any k \neq 0 • projective space P^2 = the set of equivalence classes of vectors in R^3 – (0,0,0) under the equivalence relation (a,b,c) = k(a,b,c) for nonzero k • in projective space, objects are only known up to a scalar multiple • therefore, both the point and line in 2-space have two degrees of freedom (dof), not three • the two dof in a line (a,b,c) are the two independent ratios a:b:c • projective space always uses one extra dimension • can visualize P^2 as lines through the origin in 3-space • projective geometry is the study of the geometry of projective space

20. Perspective projection is now linear • translate from homogeneous coordinates in projective space to Cartesian coordinates in Euclidean space by: • (x,y,w)  (x/w, y/w) • after all, (x,y,1)  (x,y) and (kx,ky,k)  (x,y) • P^2  R^2 • this should look familiar: perspective projection • in projective space, perspective projection is a linear map: • Euclidean: (X,Y,Z)  f/Z (X,Y) • (X,Y,Z,1)  (fX,fY,Z) • this can be encoded by a matrix • (X,Y,Z,1)  (diag(f,f,1) 0) (X,Y,Z,1) • aside: translation in 3-space is now encodable by a homogeneous 4x4 matrix

21. Duality • points and lines are dual under this representation • think of ax+by+cw = 0 • point representation and line representation are interchangeable in this way • can exchange points and lines with no ramifications • therefore, all statements about points can be replaced by statements about lines • e.g., P.L = 0: P lies on L or L intersects P • e.g., P1xP2 maps from two points to a line, and L1xL2 maps from two lines to a point • in general, points and hyperplanes in n-space are dual