CS 395/495-25: Spring 2004

1. CS 395/495-25: Spring 2004 IBMR: The Fundamental Matrix Combines Two Cameras Jack Tumblin jet@cs.northwestern.edu

2. Basic Camera P0: P3P2(or camera R3) • Basic Camera P0 is a 3x4 matrix: • Non-square pixels? change scaling (x, y) • Parallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” yc xc yc zc 1 xfspx0 0 yfpy00 0 1 0 x y z = y y’ p C z K (3x3 submatrix) zc f y' P0 X = x x' xc [K 0] = P0

3. ONE Camera Matrix P links P3P2: xfspx0 0 yfpy00 0 1 0 y Input:X 3D World Space Output: x 2D Camera Image x X (world space) z yc C zc xc x(camera space) y x • Basic camera: x = P0 X whereP0 = [K | 0] = • World-space camera: translate X (origin) to camera location C, then rotate: x = PX = (P0·R·T) X = K·R·[I C] X • Rewrite as: P = K[R -RC] • Redundant notation:P = [M p4]M = RKp4 = -K·R·C ~ ~ ~ ~

4. ONE Complete Camera Matrix P ~ ~ C y C x X (world space) z yc C zc xc y x • K matrix: “internal camera calib. matrix” • R·T matrix: “external camera calib. matrix” • T matrix: Translate world to the cam. origin • R matrix: 3D rotate world to fit cam. axes Combine: write (P0·R·T·X = x or K[R -RC]·X = x or P·X = x Input:X 3D World Space Output: x 2D Camera Image ~ x(camera space)

5. TWO Cameras: Epipolar Geometry Basic idea: • Given ONLY images from 2 cameras C, C’ • Different views of same objects X, butwe don’t know world-space points X. • If we choose an x, how can we find x’ ? • How are x, x’ linked? X x’ x C’ C

6. TWO Cameras: Epipolar Geometry Basic idea: • 2 cameras located at C, C’ in world space. • Find ‘baseline’ through camera centers C, C’ • Baseline hits image planes at ‘epipoles’ Notice baseline and Xform a plane... (OTHER planesthru baseline for other X...) X x’ x baseline C’ C

7. TWO Cameras: Epipolar Geometry Basic idea: • 2 cameras located at C, C’ in world space. • Find ‘baseline’ through camera centers C, C’ • Baseline hits image planes at ‘epipoles’ • Family of planes thru baseline are all the ‘epipolar planes’ • Image of planes = lines =‘epipolar lines’ • Lines intersect at epipolar points in both images. X x’ x baseline C’ C

8. TWO Cameras: Epipolar Geometry Summary: • Connect cameras C, C’ with a baseline, which hits image planes at epipoles e, e’. • Chose any world pt X, then  everything is coplanar! epipolar plane includes image points x, x’, and these connect to epipoles e,e’ by epipolar lines L, L’ X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

9. Epipolar Geometry Useful properties: • Every image point x maps to an epipolar line L’also: • Epipoles e,e’ = each camera’s view of the other • All epipolar lines L’ pass through epipole e’ • Epipolar Line L’ is (image of the CX ray...) • Epipolar Line L’ links (image of C) to (image of X) X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

10. Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. • Maps image point x to image point x’: x’T F x = 0 but F is only Rank 2 – given only x, F cannot find x’ for you!! • Maps image point x to epipolar line L’: F x = L’ X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

11. Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. • Maps image point x to image point x’: (x’T F) x = 0 (We know x is on line L, thus (LT) x = 0, so • Maps image point x to epipolar line L’: F x = L’ X epipolar plane  epipolar line L’ epipolar line L x’ x baseline C’ e’ e C

12. Fundamental Matrix: Fx = L’ • 2) What is F if we DON’T know the cameras P, P’,but we DO know some corresp. point pairs (x, x’)? • F finds epipolar line L’ from point x : Fx = L’ • (Recall that if (any) point x’ is on line a L’, then x’T L’= 0) • Substitute Fx for L’: x’TF x = 0 AHA! we can find F using DLT-like method! see Chap. 10 X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

13. Derive the Fundamental Matrix: • Recall Pseudo-Inverse: P+ = PT ( P PT)-1 and P P+= I (pg148) • Write world-space ray XC as: X() = P+x + C • Other camera’s image of the ray is its epipolar line L’: L’() = P’X() = P’P+x + P’C . • But P’C = e’; it is the epipole of the other camera, so L’() = P’P+x + e’ X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

14. Derive the Fundamental Matrix: • But P’C = e’; it is the epipole of the other camera, so L’() = P’P+x + e’ . • Rewrite epipolar line L’ using points P’P+x and e’ • (Recall: find line between two P2 points with cross product: L= x1 x2) • To get L’ = e’P’ P+ x or L’ = F x X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

15. Derive the Fundamental Matrix: • But P’C = e’; it is the epipole of the other camera, so L’() = P’P+x + e’ . • Rewrite epipolar line L’ using points P’P+x and e’ • (Recall: find line between two P2 points with cross product: L= x1 x2) • To get L’ = (e’) (P’ P+ x) or L’ = F x THUS F = [e’]P’ P+ X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C Hunh? What Is This?!?!

16. Derive the Fundamental Matrix: F = [e’]P’ P+But what’s this? A NEW TRICK: • Cross Product written as matrix multiply (Zisserman pg. 554) a b = = = = [a] · b • So write: a b = -b a = [a]·b = (aT·[b])T X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C b1 b2 b3 a2b3 – a3b2 a3b1 – a1b3 a1b2 – a2b1 0 -a3 a2 a3 0 -a1-a2 a1 0 b1 b2 b3 a1 a2 a3 a ‘skew symmetric’ matrix

17. Fundamental Matrix: Fx = L’ • Given a point pair (x,x’) Matrix F is unique! • Cool! still works even for different cameras! • F tied DIRECTLY to corresp. point pairs (x, x’): • F finds epipolar line L’ from point x : Fx = L’ • (Recall that if (any) point x’ is on line a L’, then x’T L’= 0) • Substitute Fx for L’: x’TF x = 0 X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C F = [e’] P’ P+

18. Fundamental Matrix: Fx = L’ 1) How do we findF? If we know the camera matrices P and P’ (we almost never do), we showed (Zisserman pg 224) F = [e’]P’ P+ X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C ( Recall: a b = -b a = [a]·b = (aT·[b])T ) (Recall: P+ = PT(PPT)-1, the pseudo-inverse)

19. Fundamental Matrix: Fx = L’ • 2) What is F if we DON’T know the cameras P, P’,but we DO know some corresp. point pairs (x, x’)? • F finds epipolar line L’ from point x : Fx = L’ • (Recall that if (any) point x’ is on line a L’, then x’T L’= 0) • Substitute Fx for L’: x’TF x = 0 AHA! we can find F using DLT-like method! see Chap. 10 X epipolar plane  epipolar line L’ epipolar line L X() x’ x baseline C’ e’ C

20. Fundamental Matrix Summary (pg. 226) F is 3x3 matrix, maps P2P2, rank 2, 7-DOF • If world space pt X image space pts. x and x’ then x’TF x = 0 • Every image pt has epipolar line in the other image:Fx = L’FTx’ = L • Baseline pierces image planes at epipoles e, e’ : Fe = 0 FTe’ = 0

21. Fundamental Matrix Summary (pg. 226) F is 3x3 matrix, maps P2P2, rank 2, 7-DOF • Given camera matrices P, P’, find F matrix by: F = [e’] P’ P+ (recall: e’ is image of C: e’ = P’C) • F is unaffected by any proj. transforms done on BOTH cameras(PH, P’H) has same F matrix as (P, P’) for any full-rank H (e.g. F measures camera C vs. Camera C’ only, no matter where you put them)

22. Fundamental Matrix Uses tx ty tz Special case: camera translate only (no rotations) • Camera matrices are P= K[ I | 0 ], P’ = K[I | t ] • where K is internal calib., t is 3D translation vector • F matrix simplifies to F = [e’] • Epipolar lines are all parallel to direction t • x,x’ displacement depends only on t & 3D depth z: x’ = x + (Kt)(1/z)

23. Fundamental Matrix Uses Can we find a camera matrix K from motion + fundamental F? • Let one camera position define the world’s coords:P = P0 = K[ I | 0 ], and define the other as P’ = [M | m] = K[R | -RC] where K is internal calib., R is rotation C is world-space position • F matrix simplifies to F = [m]M • If we know how we moved the camera (R, C matrices) then find F by correspondence, and then solve for K. (pg 237) ~ ~ ~

24. Fundamental Matrix Uses General movement? • Recall: rotations don’t change image content (camera rotate a P3 homography matrix H) • ANY cameras, ANY movements can then be warped to remove rotations, THUS • Can ALWAYS get parallel epipolar lines! • Easier to find correspondences • Easier to find depth values z • ‘Parallel Epipolar Lines’==‘Rectified Image Pair’

25. Fundamental Matrix Properties (pg. 226) Why bother with F? • Can find it from image pt. correspondences only • Works even for mismatched cameras (example: 100-year time-lapse of Eiffel tower) • Choose your own world-space coordinate system. • SVD lets us recover P, P’ camera matrices from F • (4-way ambiguity; what is front/back of C and C’?) pg 240 • BUT WE DON’T NEED TO! • Complete 2-camera mapping from worldimage • 2 images + corresponding point pairs (xi,x’i)F • Let camera coords == 3D world coords, then (xi,x’i)Xi

26. Correspondence Problem: Where Computer Vision, IBMR part ways: Few Images: (image pairs, trios…) FIND Fundamental Matrix + FIND Corresponding point pairs (x,x’) ************Versus************** Many Images: GATHER enough for P2-only work: ignore or simplify the correspondence?

27. Conclusions • P2, P3 matrix forms give elegant, principled notation for ALL image geometry • Cameras, lights, points, lines, planes, conics, quadrics, twisted cubics, … • Matrix form makes everything reversible: 3D from (2D)*! • We can find shapes from images,BUT ONLY if we have enough reliable point correspondences…And THAT’s what we usually CAN’T find!

28. END

29. Another way to describe P3 lines! • Given world-space camera position C =[a b c 1] and • a world-space direction D (found from a pixel xd, as shown on prev. slide), • even though D is a ‘point at infinity’, we can use it to make a parametric line in world space:line(t) = C + Dt =[a b c 1] + [d e f 0]t • Note that ‘t’ is position along the line, and it’s in world-space coords! • If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.