Camera Calibration

1. Camera Calibration Course web page: vision.cis.udel.edu/cv March 24, 2003  Lecture 17

2. Announcements • No class Wednesday • Homework 3 due Wednesday • Midterm on Friday • Focus on lecture material up to probability last week—use readings for depth and understanding, but I won’t go there for new topics • Definitions, some calculations (e.g., convolution)

3. Outline • Estimating the camera matrix • Least-squares • Extracting the calibration matrix • Nonlinear least-squares • Estimating radial distortion

4. 3 x 4 projective camera matrix has 11 degrees of freedom (DOF): 5 intrinsic, 3 rotation, 3 translation The Camera Matrix P • The transformation performed by a pinhole camera on an arbitrary point in world coordinates is written as:

5. Applications: Football First-Down Line courtesy of Sportvision

6. Applications: Virtual Advertising courtesy of Princeton Video Image

7. First-Down Line, Virtual Advertising: How? • The 3-D geometry of the line, advertising rectangle, etc. in world coordinates can be directly transformed into image coordinates for a given camera by projecting it with P

8. The Problem: Estimating P • Given a number of correspondences between 3-D points and their 2-D image projections Xi$ xi, we would like to determine P such that xi =PXifor all i

9. Y xi Xi Z X A Calibration Target courtesy of B. Wilburn

10. Obtaining the Points • One method • Edge detection (e.g., Canny) on image of calibration target • Fit straight lines to detected edge segments • Intersect lines to find corners • Another method • Directly search for corners using feature detection • Confirm with edge information

11. Estimating P: The Direct Linear Transformation (DLT) Algorithm • Given a number of correspondences between 3-D points and their 2-D image projections Xi$ xi, we would like to determine P such that xi =PXifor all i • This is an equation involving homogeneous vectors, so PXi and xi are only in the same direction, not strictly equal • We can specify “same directionality” by using a cross product formulation:

12. DLT Camera Matrix Estimation: Preliminaries • Let xi = (xi, yi, wi)T (remember that Xi has 4 elements) • Denoting the jth row of P by pjT (a 4-element row vector), we have:

13. DLT Camera Matrix Estimation: Step 1 • Then by the definition of the cross product, xi £PXi is given explicitly as:

14. DLT Camera Matrix Estimation: Step 2 • pjTXi=XipjT, so we can rewrite the preceding as

15. DLT Camera Matrix Estimation: Step 3 • Collecting terms, this can be written as a matrix product where 0T = (0, 0, 0, 0). This is a 3 x 12 matrix times a 12-element column vector p = (p1T, p2T, p3T)T

16. DLT Camera Matrix Estimation: Step 4 • There are only two linearly independent rows here • The third row is obtained by adding xi times the first row to yi times the second and scaling the sum by -1/wi

17. DLT Camera Matrix Estimation: Step 4 • So we can eliminate one row to obtain the following linear matrix equation for the ith pair of corresponding points: • Write this as Aip = 0

18. DLT Camera Matrix Estimation: Step 5 • We need at least 5 ½ point correspondences to solve for p • Each point correspondence yields 2 equations (the two lines of Ai) • There are 11 unknowns in the 3 x 4 homo-geneous matrix P (represented in vector form by p) • Stack Ai to get homogeneous linear system Ap = 0

19. DLT Camera Matrix Estimation: Step 6 • Minimum number of correspondences • Solve linear system exactly • 6 or more correspondences: Over-determined • Seek best solution in least-squares sense courtesy of Vanderbilt U.

20. DLT Camera Matrix Estimation: Least-Squares • Want to solve Ap = 0 • Don’t want the trivial solution p = 0 • Can arbitrarily choose scale (since it’s a homogeneous vector), so set requirement on norm kpk = 1 • This is satisfied by computing the singular value decomposition (SVD) A = UDVT (a non-negative diagonal matrix between two orthogonal matrices) and taking p as the last column of V

21. courtesy of J. Bouguet Practical Considerations for P Estimation • Should have about 30 or more correspondences for a good solution • Normalize points beforehand for better numerical conditioning • Subtract centroids, scale so that average distance from origin is p2 for 2-D points and p3 for 3-D points with transformations T, U, respectively • “Denormalize” solution P’ by applying inverse scaling, translation transformations: P = T-1P’U • Degenerate configurations of 3-D points: No unique solution • Types • Points are on union of plane and single line containing camera center • Points are on a twisted cubic (space curve) with camera • Other calibration methods avoid some of these limitations, particularly the co-planarity one

22. Extracting the Camera Calibration Matrix K from P • Remember that P = K[R j t], where the camera’s intrinsic parameters are given by

23. Anatomy of P • Let c = (c’T, 1)T be the camera center in world coordinates. This is projected to the image center, which is equivalent to Pc = 0 • Since P = K[R j t], the above equation holds when t=-Rc’, so we have P = K[R j-Rc’] • Let M denote the left 3 x 3 submatrix of P • Then M = KR and we can write P = [M j-Mc’]

24. Extracting K: Method • Consider P = [M j-Mc’] • Find camera center by solving Pc = 0 • Use same SVD approach as for Ap = 0 • Find camera orientation & internal parameters by factoring M with RQ decomposition (product of upper-triangular & orthogonal matrix) M=KR • Eliminate decomposition ambiguity by requiring K‘s diagonal entries to be positive

25. Estimating P via Nonlinear Minimization • DLT method of minimizing kApk: Efficient but not optimal • A better solution can often be obtained by finding P that minimizes a sum of squared differences: • This is a nonlinear function—we can’t express it as a matrix product • Can solve with a gradient descent method for function minimization like Levenberg-Marquardt (see link on course web page for details) • Initialize with DLT estimate

26. Correcting Radial Distortion courtesy of Shawn Becker Distorted After correction

27. Modeling Radial Distortion • Some function of distance to camera center • Approximate nonlinear distortion function with Taylor polynomial • To estimate, just include in nonlinear minimization: instead of minimizing x = PX, we write it as: