Calibration

1. Calibration Dorit Moshe

2. In today’s show • How positions in the image relate to 3D positions in the world? • We will use analytical geometry to quantify more precisely the relationship between a camera, the objects it observes, and the pictures of these objects • We start by briefly recalling elementary notions of analytical Euclidean geometry. • We then introduce the various physical parameters that relate the world and camera coordinate frames, and present as an application various methods for estimating these parameters, a process known as geometric camera calibration. • We also present along the way a linear least-squares technique for parameter estimation

3. Motivation • How positions in the image relate to 3D positions in the world? • The reconstruction of 3D image is not trivial. We have to reconstruct the third coordinate!

4. Example • Rabbit or Man? • The information lays within the 3rd coordinate Markus Raetz, Metamorphose II, 1991-92

5. 2D projections are not the “same” as the real object as we usually see everyday!

6. But…

7. Introduction • Camera calibration: estimation of the unknown values in a camera model. • Intrinsic parameters - Link the frame coordinates of an image point with its corresponding camera coordinates • Extrinsic parameters - define the location and orientation of the camera coordinate system with respect to the world coordinate system

8. Euclidean Geometry - reminder • Orthonormal coordinate frame (F) is defined by a point O in E3 and three unit vectors i, j and k orthogonal to each other

9. Transformations • FP - the coordinate vector of the point P in the frame F • Let’s consider two frames : A and B • (A) = (OA, iA, jA, kA) • (B) =(OB, iB, jB, kB) • How can we express BP as a function of AP? • Let us suppose that the basis vectors of both coordinate systems are parallel to each other, i.e., iA = iB, jA = jB and kA = kB, but the origins OA and OB are distinct • We say that two coordinate systems are separated by a pure translation,

10. Pure translation • BP = AP + BOA

11. Pure rotation • When the origins of the two frames coincide, i.e.,OA = OB = O, we say that the frames are separated by a pure rotation.

12. We define the rotation matrix

13. It means that for pure rotation BP BP

14. The inverse of a rotation matrix is equal to its transpose • Its determinant is equal to 1 – the transform preserves the volume. • Not every transformation that preserves the volume keeps the sign. For example - reflection • This ortho normal transform preserves length and angles.

15. Example (Pure rotation) • kA=kB=k • The vector iB is obtained by applying to the vector iA acounterclockwise rotation of angle θabout k.

16. Translation and rotation- rigid transformation

17. As a single matrix equation: AP BP Rotation Transformation

18. Homogenous coordinates • Add an extra coordinate and use an equivalence relation • For 3D, equivalence relation k*(X,Y,Z,T) is the same as (X,Y,Z,T) • Motivation – it will be possible to write the action of a perspective camera as a matrix

19. Homogenous/Non-Homogenous transformation for 3D point • Non-homogenous to homogenous – add “1” as the 4th coordinate: • Homogenous to non- homogenous – devide 1st 3 coordinates by the 4th

20. Homogenous/Non-Homogenous transformation for 2D point • Non-homogenous to homogenous – add “1” as the 3rd coordinate: • Homogenous to non- homogenous – devide 1st 2 coordinates by the 3rd

21. Camera calibration • Use the camera to tell you things about the world • Relationship between coordinates in the world and coordinates in the image : geometric calibration • (We will not discuss here the relationship between intensities in the world and intensities in the image : photometric camera calibration.)

22. Three coordinate systems involved • Camera: perspective projection. • Image: intrinsic/internal camera parameters • World: extrinsic/external camera parameters

23. The camera perspective equation • The coordinates (x, y, z) of a scene point P observed by a pinhole camera are related to its image coordinates (x’, y’) by the perspective equation • We have by similar triangles (x,y,z)-> (f x/z, f y/z, -f ). • Ignoring the third coordinate, we get (x,y,z)-> (f x/z, f y/z) P’ P

24. Intrinsic parameters • Relate the camera’s coordinate system to the idealized coordinate system • We can associate with a camera two different image planes: the first one is a normalized plane located at a unit distance from the pinhole. We attach to this plane its own coordinate system with an origin located at the point where the optical axis pierces it. According to the perspective eq : Perspective projection:

25. Intrinsic parameters(cont) f 1

26. Intrinsic parameters(cont) • The second is the physical retina. It is located at a distance f≠1 from the pinhole, and the image coordinates (u,v) are usually expressed in pixel units. • Pixels are usually rectangular, so the camera has two additional scale parameters k and l, and: f is a distance in meters Define :

27. Intrinsic parameters(cont) • The actual origin of the camera coordinate system is at a corner C of the retina, and not at its center. It adds two parameters u0 and v0 that define the position (in pixel units) of C0 in the retinal coordinate system.

28. Intrinsic parameters(cont) • The camera coordinate system may also be skewed, due to some manufacturing error, so the angle θbetween the two image axes is not equal to 90°.

29. Intrinsic parameters(cont) • Using homogenous coordinates: 3x4 matrix

30. Intrinsic parameters(cont) • The physical size of the pixels and the skew are always fixed for a given camera, and they can in principle be measured during manufacturing

31. Extrinsic parameters • Relate the camera’s coordinate system to a fixed world coordinate system and specify its position and orientation in space. • We consider the case where the camera frame (C) is distinct from the world frame (W). Non-homogenous coordinates Homogenous coordinates

32. Extrinsic parameters(cont)

33. Combining extrinsic and intrinsic calibration parameters • M can be defined with 11 free coefficients • 5 are intrinsic parameters – α,β,u0,v0,θ • 6 are extrinsic – the 3 angles defining R,3 coordinates of t 3 coords of t 3 raws of R M is only defined up to scale in this setting!!

34. Rewriting the equation World coordinates Pixel coordinates

35. Z is in the camera coordinate system, but we can solve it, cause And we get Relation between image positions, u,v to points at 3D positions in P (homogenous coordinates)

36. Calibration methods • Techniques for estimating the intrinsic and extrinsic parameters of a camera • Suppose that a camera observes n geometric features such as points or lines with known positions in some fixed world coordinate system. • We will: • Compute the perspective projection matrix M associated with the camera in this coordinate system • Compute the intrinsic and extrinsic parameters of the camera from this matrix Which features should we choose?

37. A linear approach to camera calibration • For each feature point, i, we have : • For n features, we will get 2n equations

38. A linear approach to camera calibration(cont) =0 P m

39. A linear approach to camera calibration(cont) • When n>=6, the system is over-constrained, i.e. there is no non-zero vector m in R12 that satisfies exactly these equations. • On the other hand, a zero vector is always a solution. • According to the linear least-squares methods, we want to compute the value of the unit vector m that minimizes |Pm|2. • In particular, estimating the vector m reduces to computing the eigenvectors and eigenvalues of the 12x12 matrix PTP

40. Linear least squares methods • Let us consider a system of n equations, p unknowns: • A is n x p matrix with coefficients aij, x = (x1,..,xp)T There is no single solution if n≥p. The non trivial solution exists only if A is non singular. We will try to find vector x that minimize E (the error measure):

41. Linear least squares methods(cont) • Need to impose a contraint on x, since x=0 yields the minimum. • Since E(λx) = λ2E(x), we will use the contraint |x2|=1. • E=xT(ATA)x , where ATA (pxp) matrix is positive smmetric matrix • It can be diagonalized in an orthonormal basis of eigenvectors ei (i=1,..,p) associated with eigenvalues 0≤ λ1≤ λ1≤ … ≤ λp • we can write x as x=μ1e1+… μnen, that (μ12 +…+ μp2 )=1  e1 minimizes the error E. It is the eigenvector associated with the minimum eigenvalue of ATA (λ1 ) :

42. Recovering the intrinsic and extrinsic parameters • Once we have the M matrix, we can recover the intrinsic and extrinsic parameters in a simple mathematical process, described in Forsyth&Ponce, section 6.3.1

43. Camera Calibration with a Single Image • Sometimes more than one view of the same picture is used to estimate calibration parameters (For example, Stereo) • Most Camera parameters can be estimated from the measurements of a single image when sufficient geometric object knowledge is available. • The object knowledge used in the approach described here consists of parallelism and perpendicularity assumptions of straight object edges. • In buildings parallel and perpendicular edges are usually abundant. Therefore, this method is often applicable for historic imagery of possibly demolished buildings taken with an unknown camera.

44. Targets for camera calibration

45. Projection of each point gives us two equations and there are 11 unknowns. 6 points in general position are sufficient for calibration.

46. The 6 anchor points clicked by the user are represented in green. If the user had clicked more accurately, they should lie exactly at the corners of the small white squares. Using these 6 points and the corresponding 3D anchor points, the program computes an initial estimate of the projection matrix. http://www-sop.inria.fr/robotvis/personnel/lucr/detecproj.html

47. We take as input a set of at least 6 non-coplanar 3D anchor points, and their 2D images. The 2D coordinates do not need to be very accurate, they are typically obtained manually by a user who clicks their approximate position.

48. Summary • We saw the goal of calibration • We mentioned Euclidean Geometry • We learned about internal/external camera parameters • We learned how to compute them from a given set of points • We saw an example of calibration by one picture only • We will see (stereo lecture) computing 3D coordinates from more than one picture (More than one view).