Auto-calibration

1. Auto-calibration • we have just calibrated using a calibration object • another calibration object is the Tsai grid of Figure 7.1 on HZ182, which can be used to solve for the camera matrix, including automatic calibration using Canny edge detection (if the grid is the only object in the image?) • we now want to calibrate without any calibration objects, instead using multiple views • recall that our main goal is to promote affine reconstructions to metric reconstructions • what is the homography that corrects the camera matrix? • auto-calibration = “computation of metric properties of the cameras and/or the scene from a set of uncalibrated images” • HZ458, Chapter 19

2. Calibration seeks a rectifying homography • consider multiple views with fixed K • suppose that we have acquired an image sequence by a camera with fixed internal parameters (fixed K; basically fixed focal length) • we are looking for the m calibrated cameras PMi (M for ‘metric’) that took the images • in our case, the motion of these cameras is arbitrary, but the calibration matrix is constant • PMi = K [Ri | ti] (where we have replaced -RC by the translation vector t) • we presently know the m uncalibrated cameras Pi, where PMi = PiH • consider the projective reconstruction (Pi, Xj) computed from these views • projective camera matrices Pi and projectively reconstructed 3D points Xj • built from fundamental matrices (which are built from point correspondences) • these camera matrices will not satisfy the constraint of constant K • goal: compute rectifying homography H that yields a metric reconstruction: • (Pi, Xj)  metric (Pi H, H-1 Xj) • the cameras of Pi H have a consistent calibration matrix K • note that the same H is used to correct all cameras • direct vs stratified methods • stratified methods get metric from affine reconstruction very quickly (linear time) • aside: sometimes a camera calibration may be found more quickly than a metric scene reconstruction (e.g., if camera just rotates) • HZ459

3. Simplifying the homography • to simplify H, we can make one global similarity (rotation, translation, scale) transform, since we are only solving up to a similarity anyway • i.e., factor out the homography’s irrelevant similarity component • assume wlog PM1 = K [I | 0] and P1 = [I | 0] • apply a similarity transform to move world frame to first camera’s frame • this is the global translation and rotation • let the homography be H = [A t; vt k] • the world frame change allows us to simplify H to [K 0; vt k] • applying PM1 = P1 H, we get A = K and t = 0 • [K 0] = [I 0] H or (since premultiplication with [I 0] ignores the 4th row) [K 0] = [A t] • now we use our scale dof to set k = 1 • H is nonsingular (nonzero det) so k \neq 0, scale the frame to make k=1 • H = [K 0; vt 1]: 8 dof (5 + 3) • v may be replaced by –Kpt where the plane at infinity in the projective reconstruction is (pt 1) • now you see the relationship between H and K (knowledge of K is enough) • HZ459-460

4. Absolute dual quadric Q*∞ (abridged) • recall π∞ and Ω∞, objects at infinity • want to motivate the importance of Q*∞for calibration to metric structure • absolute dual quadric Q*∞is: • dual of absolute conic • planes tangent to absolute conic Ω∞(a degenerate tangent space of a surface) • represented by 4x4 matrix diag(1,1,1,0) • just like Ω∞ • Q*∞= dual(Ω∞) is fixed iff map is a similarity • i.e., Q*∞characterizes the similarity • measures angle: • cos (angle between planes π1 and π2) = π1t Q*∞π2 / sqrt(π1t Q*∞π1 ) sqrt(π2t Q*∞π2) • use dual quadric Q*∞ instead of primal conic Ω∞ in calibration • just like we used the dual conic C*∞instead of (primal) circular points in 2d rectification • interesting aside: null vector of Q*∞= π∞ • HZ83-85

5. Calibration with quadrics, conics, and the calibration matrix • recall relationship between ωand K (see slide on ω): • ω = (KKt)-1 • this is a relationship between a conic and K • we want to establish a relationship between quadrics and conics, which will be combined with the relationship between conics and K • Q  C and C  K yields Q  K • then constraints on K can be transferred to Q • in particular, to the absolute dual quadric • will allow us to find the absolute dual quadric to calibrate • HZ462

6. Outlines • consider the tangent space of a surface • consider all tangent planes that contain the camera center • the silhouette of a smooth surface S = the locus of points on S whose tangent planes contain the camera center • it defines a smooth curve • called a contour generator in HZ • outline of this surface = image of the silhouette • outline = apparent contour = profile • silhouette depends on the camera center • outline depends on the camera center and the image plane • in graphics, the silhouette is important because it defines the boundary between visible and invisible parts of the surface • we shall revisit when we discuss tangential varieties, growing out of dual surfaces • HZ200

7. Spherical outlines • the silhouette of a sphere is a circle • plane of circle is orthogonal to Cc (c = sphere center) • so cone of rays from camera center C is a right cone • the outline of a sphere is a conic • intersection of image plane and cone of rays is a conic section • HZ201

8. Quadric outlines • now projectively transform 3-space • sphere becomes a quadric Q • outline becomes a conic • silhouette is a conic too (intersection of plane with Q) • therefore, outline of a quadric is a conic • Thm: Under camera matrix P, the outline of quadric Q is the conic C with dual C* = PQ*Pt • proof: tangent lines L of C back-project to tangent planes π = PtL of Q  πt Q* π = 0  (LtP) Q* (PtL) = 0  Lt (PQ*Pt)L = 0  PQ*Pt is dual conic of C • note that this relates the camera, quadric and conic • HZ201

9. Exploiting mutual bonds to calibrate • C* = PQ*Pt relates camera, quadric and conic • apply it to the absolute conic and the absolute dual quadric • fact: Q*∞ projects to ω • new bond ω Q*∞ • old bond ω K • this creates another new bond Q*∞ K • also have bond K  H • we use this bond to solve for Q*∞ from constraints on K • then the calibrating homography may be determined from Q*∞ (diag(1,1,1,0)  H diag(1,1,1,0) Ht = Q*∞) • metric reconstruction: apply H-1 to points and H to cameras • summary: K constraints  Q*∞ homography • HZ462-3 (including algorithm 19.1)

10. Solving for Q*∞ • build the matrix ω* = KKt (e.g., when skew = 0) • Table 19.1 • the zero entries of this matrix generate constraints on Q*∞ from ωi* = Pi Q*∞Pit = 0 • e.g., known principal point in image (which is then moved to origin) yields two constraints ω*13 = ω*23 = 0 • known principal point: 2; also zero skew: 3; also known aspect ratio: 4 (all linear) • zero skew only: 1 quadratic constraint • need to find 10 constraints to solve for Q*∞ • need to find 8 constraints to solve for Q*∞ if we use det Q*∞ = 0 as 9th yielding a quartic equation that has 4 potential solutions • use same approach as the 7-point algorithm for F • what will yield 8 or 10 constraints? • camera where only focal length is unknown (principal point known, pixel aspect ratio known, zero skew) yields 4 constraints per image, so 2 images are enough • this is the common case • known principal point only: 2 constraints per image, so need 4-5 images • see Table 19.3 HZ469 for other scenarios and the required # of views • this completes calibration • HZ464-466