1 / 62

Projective Camera Matrix: Formulating the 3D to 2D Mapping

This article explores the concepts of projective cameras and affine cameras, discussing the formalization of the 3D to 2D mapping using homogeneous coordinates. It also explains the basic camera matrix and how it is composed and controlled as a matrix product.

calebb
Download Presentation

Projective Camera Matrix: Formulating the 3D to 2D Mapping

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 395/495-25: Spring 2004 IBMR: R3R2 and P3P2: The Projective Camera Matrix Jack Tumblin jet@cs.northwestern.edu

  2. Admin Reminders • Projects Progress: Reading Week Signup… • Can you explain how Debevec finds: • Camera response curve? • One single unified floating-point image from multiple photos taken with different exposures? • No? For next class,write down the steps you DO understand…

  3. Cameras Revisited • Goal: Formalize projective 3D2D mapping • Homogeneous coords handles infinities well • Projective cameras ( convergent ‘eye’ rays) • Affine cameras (parallel ‘eye’ rays) • Composed, controlled as matrix product • But First: Cameras as Euclidean R3R2: x’ = fx / zy’ = fy / z y y’ C z Review Trucco & Verri: End of Chapter 2… f

  4. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Point • Principal Axis • Principal Plane (?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc zc xc

  5. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Point • Principal Axis • Principal Plane (?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc zc f xc

  6. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Point • Principal Axis • Principal Plane (?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc C f zc xc

  7. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Point p • Principal Axis • Principal Plane (?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc C p f zc xc

  8. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Pointp • Principal Axis • Principal Plane (?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc C p f zc xc

  9. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Pointp • Principal Axis • Principal Plane (?!?! DOESN’T touch principle point P !?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc C p f zc xc

  10. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Pointp • Principal Axis • Principal Plane(?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc p zc f C =[0,0,0] xc

  11. Cameras Revisited Plenty of Terminology: • Image Plane or Focal Plane • Focal Distance f • Camera Center C • Principal Pointp • Principal Axis • Principal Plane(?!?! DOESN’T touch principle point!?!?) • Camera Coords (xc,yc,zc) • Image Coords (x’,y’) yc C p f zc y' x' xc

  12. Cameras Revisited • Goal: Formalize projective 3D2D mapping • Homogeneous coords handles infinities well: • Projective cameras ( convergent ‘eye’ rays) • Affine cameras (parallel ‘eye’ rays) • Composed, controlled as matrix product • Recall Euclidian R3R2 : x’ = fx / zy’ = fy / z (Much Better: P3P2) y y’ C z f Review Trucco & Verri: End of Chapter 2…

  13. Most-Basic Camera • Instead of x’ = fx / z (As in Trucco & Verri Chap. 2)y’ = fy / z • try this 3x4 matrix: • Note!X is an augmented R3 vector: ? OK for P3 ? ?isx in P2 space? • As shown: image origin (x’,y’) is principal point p, but pixel counts usually start at an image corner… yc y xc yc zc 1 y’ f 0 0 0 0 f 0 00 0 1 0 x’ y’ z’ p C = z zc f y' x' xc P0 X = x

  14. Most Basic Camera P0 • Basic Camera as a 3x4 matrix: • To translate image origin (x’,y’) away from zc axis:Shift principal point p from (0,0) to (px,py) • does NOT use ‘homogeneous’ 1 term in X • is NOT obvious: scales zc (think in P2 zc=x3 = 1,...) yc xc yc zc 1 f 0 px0 0 f py00 0 1 0 x’ y’ z’ = y y’ p C z zc f y' P0 X = x x' xc

  15. Basic Camera P0: P3P2(or camera R3) • Basic Camera P0 is a 3x4 matrix: • Non-square pixels? change scaling (x, y) • Parallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” yc xc yc zc 1 xfspx0 0 yfpy00 0 1 0 x y z = y y’ p C z K (3x3 submatrix) zc f y' P0 X = x x' xc [K 0] = P0

  16. Complete Camera Matrix P y x X (world space) z yc C zc xc y x • K matrix: “internal camera calib. matrix” • R·T matrix: “external camera calib. matrix” • T matrix: Translate world to cam. origin • R matrix: 3D rotate world to fit cam. axes Combine: write (P0·R·T)·X = x as P·X = x Input:X 3D World Space Output: x 2D Camera Image x(camera space)

  17. The Pieces of Camera Matrix P yc p C f zc xc • • ••• • ••• • • • xw yw zw tw Columns of P matrix = image of world-space axes: • p1,p2,p3 == image of x,y,z axis vanishing points • Direction D = [1 0 0 0]T = point on P3’s x1 axis, at inifinity • PD = 1st column of P = P1. Repeat for y and z axes. • p4 == image of the world-space origin pt. • Proof: let D = [0 0 0 1]T = direction to world origin • PD = 4th column of P = image of origin pt. • • • •• • • •• • • • xc yc zc p1 p3 p4 P p2 P = P·X = x, or =

  18. The Pieces of Camera Matrix P yc p C f zc xc xw yw zw tw Rows of P matrix: camera planes in world space • row 1 = P1T = image x-axis plane • row 2 = P2T = image y-axis plane • row 1 = P3T = camera’s principal plane P1T • • • •• • • •• • • • • • ••• • ••• • • • xc yc zc P P·X = x, or = P = P2T P3T

  19. The Pieces of Camera Matrix P xw yw zw tw Rows of P matrix: planes in world space • row 1 = P1 = world plane whose image is x=0 • row 2 = P2 = world plane whose image is y=0 • row 3 = P3 = camera’s principal plane P1T • • • •• • • •• • • • • • ••• • ••• • • • xc yc zc P P·X = x, or = P = P2T P3T yc Why? Recall that in P3, point X is on plane  if and only if T·X = 0, so… p C f zc xc

  20. The Pieces of Camera Matrix P xw yw zw tw Rows of P matrix: planes in world space • row 1 = P1 =world plane whose image is x=0 • row 2 = P2 = world plane whose image is y=0 • row 3 = P3 = camera’s principal plane P1T • • • •• • • •• • • • • • ••• • ••• • • • xc yc zc P P·X = x, or = P = P2T P3T yc p C f zc xc

  21. The Pieces of Camera Matrix P xw yw zw tw Rows of P matrix: planes in world space • row 1 = P1 = image’s x=0 plane • row 2 = P2 = image’s y=0 plane • Careful! Shifting theimage origin by px, pyshifts the x=0,y=0 planes! P1T • • • •• • • •• • • • • • ••• • ••• • • • xc yc zc P P·X = x, or = P = P2T P3T yc zc C y' p f x' xc

  22. The Pieces of Camera Matrix P xw yw zw tw Rows of P matrix: planes in world space • row 1 = P1 = image’s x=0 plane • row 2 = P2 = image’s y=0 plane • row 3 = P3 = camera’s principal plane P1T • • • •• • • •• • • • • • ••• • ••• • • • xc yc zc P P·X = x, or = P = P2T P3T yc p C f zc xc

  23. The Pieces of Camera Matrix P xw yw zw tw Rows of P matrix: planes in world space • row 1 = P1 = image x-axis plane • row 2 = P2 = image y-axis plane • row 3 = P3 = camera’s principal plane • princip. plane P3= [p31 p32 p33 p34]T • its normal direction: [p31 p32 p33 0 ]T • Why is it normal? It’s the world-space P3 direction of the zc axis P1T • • • •• • • •• • • • • • ••• • ••• • • • xc yc zc P P·X = x, or = P = P2T P3T Principal plane P3 yc zc p C f xc

  24. The Pieces of Camera Matrix P xw yw zw tw • Principal Axis Vector (zc) in world space: • Normal of principal plane: = m3 = [p31 p32 p33 ]T • P3 Scaling  Ambiguous direction!! +/- m3 ? • Solution: use det(M)·m3as front of camera • Principal Point p in image space: • image of (infinity point on zc axis = m3)M·m3 = p = x0 (Zisserman book renames p as x0) • • ••• • ••• • • • • • • •• • • •• • • • m1T xc yc zc P p4 P·X = x, or P = = m2T m3T M yc p x0 C f zc xc

  25. The Pieces of Camera Matrix P xw yw zw tw Where is camera in world space? at C: • Camera center C is at camera origin(xc,yc,zc)=0=C • Camera P transforms world-space point C to C=0 • But how do we find C ? It is the Null Space of P :P C = C = 0 (solve for C. SVD works, but here’s an easier way:) Camera’s Position in the World: C = • • ••• • ••• • • • • • • •• • • •• • • • m1T xc yc zc P p4 P·X = x, or P = = m2T m3T ~ M ~ ~ ~ ~ yc • • • 1 ~ -M-1·p4 C zc p f xc

  26. Uses for Camera Matrix P y x z xw yw zw tw Each world space R3 point d = (x,y,z)defines a world-space P3directionD = (x,y,z,0). What is image point xd from direction D? xd = PD = [M | p4] D = Md (P4 column has no effect, because of D’s zero): xd = Md or M-1xd = d • • ••• • ••• • • • • • • •• • • •• • • • xc yc zc P M p4 P·X = x, or P = = yc xd d ~ p C f zc X (world space) xc

  27. Uses for Camera Matrix P xw yw zw tw Given image pointx0 and camera matrix P,Find ray X() in world space through both: Slow, Obvious way: ‘Pseudo-invert’ P, apply to x0: • Define pseudo-inverse P+ as = PT(PPT)-1 (note P·P+ = I) • Find a world-space point on ray: X0 = P+ x0 • LIRP with world-space camera point C: X() = C + (X0-C) • • ••• • ••• • • • • • • •• • • •• • • • xc yc zc P M p4 P·X = x, or P = = X() yc x0 ~ ~ ~ p ~ C f zc xc

  28. Uses for Camera Matrix P xw yw zw tw Given image pointx0 and camera matrix P,Find ray X() in world space through both: Better way: ray from C to point D0 at infinity: • Point x0 is the image of (unknown) world-space directionD0 = (x,y,z,0)T. Define a point d0 = (x,y,z)T. • Recall: we can find d0 from the image: M-1x0 = d0 • Recall: world-space camera position C = -M-1p4 X() =  M-1x0 + C = M-1(x0 – p4) 0 1 1 • • ••• • ••• • • • • • • •• • • •• • • • xc yc zc P M p4 P·X = x, or P = = ~ X() yc ~ x0 ~ ~ p C f zc xc

  29. Uses for Camera Matrix P xw yw zw tw Given world-space pointX0, camera matrix P,Find camera depth z0: • X0 = [xw, yw, zw, tw]Tseen thru camera P is X0·P = x0 = [xc, yc, 1]T·wc • Then signed depth z0 is: z0 = wc sign(det M)tw || m3 || • • ••• • ••• • • • • • • •• • • •• • • • m1T xc yc zc P p4 P·X = x, or P = = m2T m3T M X0 yc x0 zc z0 p C f xc

  30. Optional (Skipped): ~ • P = [K|0]·R·T = K [ R | -RC ]How can we separate K, R? • Answer: K is triangular; use QR decomposition • Cameras at Infinity: • Orthographic or ‘Parallel Projection’ Cameras • Transition to Orthographic: • Weak Perspective projection cameras • the ‘zoom’ lens (variable f) • Moving line-scan or ‘pushbroom’ cameras • Translation Scan: aerial/sattelite cameras • Cylindrical Scan: panoramic cameras • UNC ‘HiBall Tracker’: 6 tiny self-locating line-scan cameras

  31. Chapter 6 In Just One Slide: Given point correspondence sets (xi Xi), How do you find camera matrix P ? (full 11 DOF) Surprise! You already know how ! • DLT method: -rewrite H x = x’ as Hx  x’ = 0 -rewrite P X = x as PX  x = 0 -vectorize, stack, solve Ah = 0 for h vector -vectorize, stack, solve Ap = 0 for p vector • More data  better results (at least 28 point pairs)(why so many? rule-of-thumb: #constraints = 5x #DOF = 55 = 27.5 point pairs) • Algebraic & Geometric Error, Sampson Error…

  32. END

  33. Camera Matrix P links P3P2 xfspx0 0 yfpy00 0 1 0 ~ C  C Input:X 3D World Space Output: x 2D Camera Image y x X (world space) z yc C zc x(camera space) xc y x • Basic camera: x = P0 X whereP0 = [K | 0] = • World-space camera: translate X (origin) to camera location C, then rotate: x = PX = (P0·R·T) X • Rewrite as: P = K[R | -RC] • Redundant notation:P = [M | p4]M = RKp4 = -K R C ~ ~ ~

  34. More One-Camera Details Full 3x4 camera matrix P maps P3world to P2 image ? What does it do to basic 3D world shapes? • 3-D Point: • Take any 3D world-space point X, • Coord. systems change: let X be Xp = (xp,yp,0); • Matrix P reduces to 3x3 matrix H in P2: xp = P·Xp = • THUSAny P3point becomes a P2point xpyp0tp xpyptp h11 h12 h13 h21 h22 h21 h31 h32 h33 p11 p12 p13 p14 p21 p12 p23 p24 p31 p32 p33 p34 =

  35. More One-Camera Details Full 3x4 camera matrix P maps P3world to P2 image ? What does it do to basic 3D world shapes? • 3-D Plane • Given any point X on a plane  in P3, • Coord. systems change: let X be X = (x,y,0); • Matrix P reduces to 3x3 matrix H in P2: x = P·X = • THUSAny P3plane becomes a P2plane xy0t xyt h11 h12 h13 h21 h22 h21 h31 h32 h33 p11 p12 p13 p14 p21 p12 p23 p24 p31 p32 p33 p34 =

  36. More One-Camera Details y x z Full 3x4 camera matrix P maps P3world to P2 image ? What does it do to basic 3D world shapes? • 3D Directions and image Points: World-space P3 direction D image space point xd: Recall directionD = (x,y,z,0)(a point at infinity) sets a R3 finite point d = (x,y,z). xd = PD = [M | p4] D = Md p4column has no effect, because of D’s zero;Recall M = KR xd = Md M-1xd = d D xd yc d ~ p C f zc X (world space) xc

  37. More One-Camera Details Full 3x4 camera matrix P maps P3world to P2 image ? What does it do to basic 3D world shapes? Forward Projection: • Line / Ray in world Line/Ray in image: • Ray in P3 is: X() = A + B • Camera changes to P2: x() = PA + PB yc A PA B p C f zc xc

  38. More One-Camera Details Full 3x4 camera matrix P maps P3world to P2 image ? What does it do to basic 3D world shapes? Back Projection: • Line L in image Plane L in world: • Recall: Line L in P2 (a 3-vector): L =[x1 x2 x3]T • Plane L in P3 (a 4-vector) is easy: L = PT·L = • (Why? Look back at bits and pieces of P…) L p11 p21 p31 p12 p22 p32p13 p23 p33p14 p24 p34 x1x2x3 yc L zc p C f xc

  39. More One-Camera Details xfspx 0 yfpy0 0 1 xfs(px-tx) 0 yf(py-ty)0 0 1  0 0 0  0 0 0 1 Full 3x4 camera matrix P maps P3world to P2 image ?What if the image plane moves? A) Translation: • Given internal camera calibration K = • (xc, yc) changes affect px,py and (zc) changes affect the focal length f: let  = (f+tz)/f, then: K’ = • Focal length change • Image coords change yc p c f zc xc

  40. More One-Camera Details xfspx 0 yfpy0 0 1 xfs(px-tx) 0 yf(py-ty)0 0 1 k 0 0 0 k 0 0 0 1 Full 3x4 camera matrix P maps P3world to P2 image ? What if the image plane moves? A) Translation: • Given internal camera calibration K = • (xc, yc) changes affect px,py and (zc) changes affect the focal length f: let k = (f+tz)/f, then: K’ = • Define effect of K’ on image points x,x’; x = [K | 0]X; x’ = [K’ | 0]X x’ = (K’ K-1) x yc p c f zc xc

  41. More One-Camera Details xfspx 0 yfpy0 0 1 Full 3x4 camera matrix P maps P3world to P2 image ?What if the image plane moves? B) Rotation: • Given internal camera calibration K = • Rotate basic camera’s output:about its center Cusing 3D rotation matrix R (3x3):x = [K | 0]X; x’ = [KR | 0] X x’ = [KR(K-1K) | 0] X = (KRK-1) [K|0] X • Get new points x’ from old image pts x (K·R·K-1) x = x’ • aka ‘conjugate rotation’; use this to construct planar panoramas yc R p c f zc xc

  42. More One-Camera Details • ALL cameras gather the same image content!Same Center Point? Same image, just rearranged! (just a planar reprojection in P3) • The camera center Cmust move to change the image content:no zooming, warping, rotation can change this! • Gathering 3-D image data requires camera movement. Full 3x4 camera matrix P maps P3world to P2 image THUS: if the image plane moves: just rearranges the image points: a) Translations: b) Rotations: (K·R·K-1) x = x’ (K’ K-1) x = x’

  43. Cameras as Protractors • Define world-space direction d: • From a P3 ideal point D = [xd yd zd 0]T define the ‘direction’ d == [xd yd zd] • D’s image point is xd = Md (recall M=KR) • Link direction d to image pt. x=(xc,yc,zc):PXd = K[R | 0] Xd =KR d = x • Ray thru image pt. x has direction d = (KR)-1x

  44. Cameras as Protractors • Angle between C and 2 image points x1,x2(see book pg 199) cos = x1T (K-TK-1) x2(x1T (K-TK-1) x1)(x2T (K-TK-1) x2) • Image line L defines a plane L • (Careful! P3 world =P2 camera axes here!) • Plane normal direction:n = KT L d2 n x2  d1 x1 C L

  45. Cameras as Protractors • Angle between C and 2 image points x1,x2(see book pg 199) cos = x1T (K-TK-1) x2(x1T (K-TK-1) x1)(x2T (K-TK-1) x2) • Image line L defines a plane L • (Careful! P3 world =P2 camera axes here!) • Plane normal direction:n = KT L d2 n Something Special here? Yes! x2  d1 x1 C L

  46. Cameras as Protractors What is (K-TK-1) ? • Recall P3 Conic Weirdness: • Plane at infinity  holds all ‘horizon points’ D (‘wrapper around the universe’) • Absolute Conic  isimaginary outermost circle of  • for ANY camera,Translation won’t change ‘Horizon point’ images: P D = x = KRd (pg200) • Absolute conic is inside ; it’s all ‘horizon points’ for ANY camera, (KR)-T(KR)-1=(K-TK-1) == ‘Image of Absolute Conic’

  47. Cameras as Protractors x1 x2 x3 0 x1 x2 x30 P () = (K-TK-1). OK.Now what was  again? • Absolute Conic = all imaginary points found on  • Satisfies BOTH x12 + x22 + x32 = 0 AND x42 = 0 • Finds right angles-- if D1D2, then: D1T· ·D2= 0 (Recall D==[x,y,z,0]) • Dual of absolute conic is Dual QuadricQ*= all imaginary planes to, and tangent to Q* • Finds right angles-- if 12, then: 1T· Q*·2= 0 • can writeor Q* as the same matrix: 1 0 0 00 1 0 00 0 1 00 0 0 0

  48. Why do we care? P  = (K-TK-1) =  = ‘Image of Absolute Conic’ • IAC is a ‘magic tool’ for camera calibration K • Recall let us find H from perp. lines. • Much better than ‘vanishing pt.’ methods • With IAC, find P matrix from an image of just 3 or more (non-coplanar) squares…Teo’s camera calibration method

  49. Summary: Cameras as Protractors D2 cos = x1T (K-TK-1) x2(x1T (K-TK-1) x1) (x2T (K-TK-1) x2) n x2  D1 x1 C L • World-Space Direction: D = [xc, yc, zc, 0]T • Direction from image point x: d = (KR)-1x • Point C and points x1,x2 form angle : (pg 199) (K-TK-1)=  = ‘Image of Absolute Conic’ Line L makes plane with normal n = KT L (in camera coords)

  50. Movement Detection? • Can we do it from images only? • 2D projective transforms often LOOK like 3-D; • External cam. calib. affects all elements of P • YES. Camera moved if-&-only-ifCamera-ray points (CxX1,X2,…) will map to LINE (not a point) in the other image • ‘Epipolar Line’ == l’ = image of L • ‘Parallax’ == x1’x2’ vector X2 X1 x2’ x1’ L x C’ l’ C

More Related