Camera calibration

1. Camera calibration Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz,, Fred Pighin and Marc Pollefyes

2. Outline • Camera projection models • Camera calibration • Nonlinear least square methods • A camera calibration tool • Applications

3. Camera projection models

4. Pinhole camera

5. Pinhole camera model (X,Y,Z) P origin p (x,y) principal point (optical center)

6. Pinhole camera model principal point

7. Pinhole camera model principal point

8. Principal point offset principal point intrinsic matrix only related to camera projection

9. Intrinsic matrix • non-square pixels (digital video) • skew • radial distortion Is this form of K good enough?

10. Distortion • Radial distortion of the image • Caused by imperfect lenses • Deviations are most noticeable for rays that pass through the edge of the lens No distortion Pin cushion Barrel

11. Camera rotation and translation extrinsic matrix

12. Two kinds of parameters • internal or intrinsic parameters such as focal length, optical center, aspect ratio:what kind of camera? • external or extrinsic (pose) parameters including rotation and translation:where is the camera?

13. Other projection models

14. Orthographic projection • Special case of perspective projection • Distance from the COP to the PP is infinite • Also called “parallel projection”: (x, y, z) → (x, y) Image World

15. Other types of projections • Scaled orthographic • Also called “weak perspective” • Affine projection • Also called “paraperspective”

16. Illusion

17. Illusion

18. Fun with perspective

19. Perspective cues

20. Perspective cues

21. Fun with perspective Ames room Ames video BBC story

22. Forced perspective in LOTR

23. Camera calibration

24. Camera calibration • Estimate both intrinsic and extrinsic parameters. Two main categories: • Photometric calibration: uses reference objects with known geometry • Self calibration: only assumes static scene, e.g. structure from motion

25. Camera calibration approaches • linear regression (least squares) • nonlinear optimization

26. Chromaglyphs (HP research)

27. Camera calibration

28. Linear regression

29. Linear regression • Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

30. Linear regression

31. Linear regression

32. Linear regression Solve for Projection Matrix M using least-square techniques

33. Normal equation Given an overdetermined system the normal equation is that which minimizes the sum of the square differences between left and right sides

34. Linear regression • Advantages: • All specifics of the camera summarized in one matrix • Can predict where any world point will map to in the image • Disadvantages: • Doesn’t tell us about particular parameters • Mixes up internal and external parameters • pose specific: move the camera and everything breaks • More unknowns than true degrees of freedom

35. Nonlinear optimization • A probabilistic view of least square • Feature measurement equations • Probability of M given {(ui,vi)} P

36. Optimal estimation • Likelihood of M given {(ui,vi)} • It is a least square problem (but not necessarily linear least square) • How do we minimize L? L P

37. We could have terms like in this Optimal estimation • Non-linear regression (least squares), because the relations between ûi and ui are non-linear functions of M • We can use Levenberg-Marquardt method to minimize it unknown parameters known constant

38. Nonlinear least square methods

39. Least square fitting number of data points number of parameters

40. Linear least square fitting y t

41. model parameters Linear least square fitting y t

42. Linear least square fitting y model parameters t

43. prediction residual Linear least square fitting y model parameters t

44. prediction residual is linear, too. Linear least square fitting y model parameters t

45. Nonlinear least square fitting model parameters residuals

46. It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum. Function minimization Least square is related to function minimization.

47. Function minimization

48. Quadratic functions Approximate the function with a quadratic function within a small neighborhood

49. Quadratic functions A is positive definite. All eigenvalues are positive. For all x, xTAx>0. negative definite A is singular A is indefinite

50. Function minimization Why? By definition, if is a local minimizer, is small enough