CS 223-B Lecture 1

1. AQUEOUS HUMOR CORNEA CS 223-BLecture 1 Sebastian Thrun Gary Bradski http://robots.stanford.edu/cs223b/index.html

2. Readings • Computer Vision, Forsyth and Ponce • Chapter 1 • Introductory Techniques for 3D Computer Vision, Trucco and Verri • Chapter 2

3. Lenses and Cameras* -- Brunelleschi, XVth Century * Slides, where possible, stolen with abandon, many this lecture from Marc Pollefeys comp256, Lect 2

4. Distant objects appear smaller A “similar triangle’s” approach to vision. Notes 1.1 Marc Pollefeys

5. Consequences: Parallel lines meet • There exist vanishing points Marc Pollefeys

6. Vanishing points VPL H VPR VP2 VP1 Different directions correspond to different vanishing points VP3 Marc Pollefeys

7. Implications For Perception* Same size things get smaller, we hardly notice… Parallel lines meet at a point… * A Cartoon Epistemology: http://cns-alumni.bu.edu/~slehar/cartoonepist/cartoonepist.html

8. Implications For Perception 2 Logrithmic in nature Perception must be mapped to a space variant grid Steve Lehar

9. The Effect of Perspective

10. Different Projections: Affine projection models: Weak perspective projection Smoosh everything flat onto a parallel plane at distance z0 is the magnification. When the scene relief is small compared its distance from the Camera, m can be taken constant: weak perspective projection. Marc Pollefeys

11. Affine projection models: Orthographic projection When the camera is at a (roughly constant) distance from the scene, take m=1. Marc Pollefeys

12. Limits for pinhole cameras Marc Pollefeys

13. Dtn1 a1 a1 a b q1 F q2 z2 d e Dtn2 a2 Notes 1.2 On to Thin Lenses … Snell’s law n1 sina1 = n2 sin a2

14. Paraxial (or first-order) optics Snell’s law: n1 sina1 = n2 sin a2 Small angles: n1a1 n2a2 Sin a  a = y/r Tan b  b = y/x Marc Pollefeys

15. Thin Lenses spherical lens surfaces; incoming light  parallel to axis; thickness << radii; same refractive index on both sides 8 Notes 1.3 z-> Marc Pollefeys

16. Thin Lenses summary Marc Pollefeys http://www.phy.ntnu.edu.tw/java/Lens/lens_e.html

17. The depth-of-field  Marc Pollefeys

18. yields Similar formula for The depth-of-field  Marc Pollefeys

19. The depth-of-field decreases with d, increases with Z0 strike a balance between incoming light and sharp depth range. Notes 1.4  Marc Pollefeys

20. Deviations from the lens model 3 assumptions : • all rays from a point are focused onto 1 image point • Remember thin lens small angle assumption 2. all image points in a single plane 3. magnification is constant Deviations from this ideal are aberrations  Marc Pollefeys

21. Aberrations 2 types : 1. geometrical 2. chromatic geometrical : small for paraxial rays study through 3rd order optics chromatic : refractive index function of wavelength  Marc Pollefeys

22. Geometrical aberrations • spherical aberration • astigmatism • distortion • coma aberrations are reduced by combining lenses  Marc Pollefeys

23. Spherical aberration rays parallel to the axis do not converge outer portions of the lens yield smaller focal lenghts  Marc Pollefeys

24. Astigmatism Different focal length for inclined rays Marc Pollefeys

25. Distortion magnification/focal length different for different angles of inclination pincushion (tele-photo) barrel (wide-angle) Can be corrected! (if parameters are know) Marc Pollefeys

26. Coma point off the axis depicted as comet shaped blob Marc Pollefeys

27. Chromatic aberration rays of different wavelengths focused in different planes cannot be removed completely sometimes achromatization is achieved for more than 2 wavelengths  Marc Pollefeys

28. Vignetting Marc Pollefeys

29. Calibration Gist: Invert the image formation process Y External coordinate system X kthcollection of points i Z Pik Note that rotation matrix R has constraints: determinant is 1, inverse is equal to transpose, optimization routine should make use of this. Then we want the actual projection to be as close as possible to The point given by the projection operator: over all i points and over all k images of grids: Rk,Tk Image plane Extrinsic Params Rotation & Translation to image frame coord. system pik • This is typically solved through a gradient decent optimization since the problem is manifestly convex. • Note that we need a good starting guess for the initial “correct” projection points p’I the optimization then iterates to solution. • Stereo would then just double the parameters adding left l and right r subscripts and additional summations over r & l. z Camera 0 x f, c, a, k Intrinsic Params focus center of image Skew a = 0 k radial and tangential distortion y (the camera will get several (K) views of this grid in rotation)

30. Assumed Perspective Projection

31. Assumed Perspective Projection

32. Cameras we consider 2 types : 1. CCD 2. CMOS  Marc Pollefeys

33. CCD separate photo sensor at regular positions no scanning charge-coupled devices (CCDs) area CCDs and linear CCDs 2 area architectures : interline transfer and frame transfer photosensitive storage  Marc Pollefeys

34. The CCD camera Marc Pollefeys

35. CMOS Foveon 4k x 4k sensor 0.18 process 70M transistors Same sensor elements as CCD Each photo sensor has its own amplifier More noise (reduced by subtracting ‘black’ image) Lower sensitivity (lower fill rate) Uses standard CMOS technology Allows to put other components on chip ‘Smart’ pixels Marc Pollefeys

36. CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill rate Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components Marc Pollefeys

37. Colour cameras We consider 3 concepts: • Prism (with 3 sensors) • Filter mosaic • Filter wheel … and X3 Marc Pollefeys

38. Prism colour camera Separate light in 3 beams using dichroic prism Requires 3 sensors & precise alignment Good color separation Marc Pollefeys

39. Prism colour camera Marc Pollefeys

40. Filter mosaic Coat filter directly on sensor Demosaicing (obtain full colour & full resolution image) Marc Pollefeys

41. Filter wheel Rotate multiple filters in front of lens Allows more than 3 colour bands Only suitable for static scenes Marc Pollefeys

42. Prism vs. mosaic vs. wheel approach # sensors Separation Cost Frame rate Artifacts Bands Prism 3 High High High Low 3 High-end cameras Mosaic 1 Average Low High Aliasing 3 Low-end cameras Wheel 1 Good Average Low Motion 3 or more Scientific applications Marc Pollefeys

43. new color CMOS sensorFoveon’s X3 smarter pixels better image quality Marc Pollefeys

44. Reproduced by permission, the American Society of Photogrammetry and Remote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry, Thompson, Radlinski, and Speert (eds.), third edition, 1966. The Human Eye Cross section of the eye Looking down the optical axis of the eye

45. Sensors and image processing Light Light RGB + B/W happens here Question: Which way does the light enter?

46. Eye cross section

47. The distribution of rods and cones across the retina Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc. Rods and cones in the periphery Cones in the fovea Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc.

48. There’s a lot more going on in Vision …i.e. Light and Surfaces

49. Real vision includes invisible inference

50. Real vision includes invisible inference