Lecture 2 Imaging Geometry and OpenCV

1. Lecture 2Imaging Geometry and OpenCV Slides by: David A. Forsyth Clark F. Olson Jean Ponce Linda G. Shapiro

2. Pinhole Cameras • Abstract camera model - box with a small hole in it • Pinhole cameras work in practice

3. Distant objects are smaller

4. Parallel lines meet Common to draw film plane in front of the focal point. Moving the film plane merely scales the image.

5. The equation of projection • Cartesian coordinates: • We have, by similar triangles, that: • (X, Y, Z) ~ (f X/Z, f Y/Z, f) • f is called the focal length. [X, Y, Z] [fX/Z, fY/Z]

6. The reason for lenses We won’t worry much about lenses in this class.

7. Lens distortion • “Barrel distortion” of rectangular grid is common for inexpensive lenses. • Precision lenses can be expensive. • Zoom lenses often show severe distortion. • Fish-eye lenses also have severe distortion.

8. Image capture • Images are not continuous • Typically captured with a CCD camera (charge-coupled-device) • The amount of light striking each location on a grid is integrated over some time period • Rows are read out one at a time • For color images, successive • pixels usually correspond to • different colors • High quality color cameras use • a beam splitter and 3 separate • CCD chips • APS (active pixel sensor) is a • cheaper (lower quality) technology

9. Resolution • Resolution often (but not always) refers to the number of pixels in the image. • Lower resolution has fewer pixels. • Interestingly, faces of people you know can usually be recognized at 64x64 (or less) pixels. • Squint and look at the lowest resolution image.

10. Some of my research • Time permitting, the following slides give a very brief overview of some of my previous and current research.

11. Visual Terrain Mapping for Mars Exploration

12. Pose Sampling for Efficient Model-Based Recognition Careful sampling of the viewpoints reduces the complexity from cubic to linear in the number of craters.

13. Robust Registration of Aerial Image Sequences Registration results Goal: Provide registration between imagesand maps for persistent aerial surveillance

14. Simple and Efficient Projective Clustering • Projective clustering problems have the following properties: • Many dimensions - d • Many points – n • Clusters do not form in the full d-dimensional space • No labeled exemplars

15. Keypoint recognition A popular object recognition technique uses descriptive “keypoints” that have been extracted from images. We are investigating getting better use out of the color information in the image when creating keypoint descriptors.

16. Image classification • Current techniques for image classification cluster image keypoints into “visual words” similar to text retrieval methods. • We are studying the use of projective clustering to improve performance. http://www.sccs.swarthmore.edu/users/09/btomasi1/tagging-products.html

17. Programming in OpenCV In OpenCV, images are represented as matrices (as in linear algebra). Mat image = imread("photo.jpg"); // Most generic declaration The image could have a number of underlying data types for each pixel: • uchar – unsigned byte (greyscale image) • Vec3b – vector of 3 bytes (color image) • Point2f – point in two dimensions, float • many others…

18. Creating images Images can be created using a number of methods: • using namespace cv; // all my code assumes this • Mat image; // creates 0x0 image • Mat image = … // uses copy constructor • Mat image(rows, cols, type); // type is CV_8U, for example • Mat image(rows, cols, type, scalarValue); • Example: Mat allBlue(360, 480, CV_8UC3, Scalar(255, 0, 0)); • Mat_<Vec3b> colorImage = imread(“color.jpg”); // Can be convenient, but now limited to Vec3b images (matrices) // Also, must declare as a similar parameter type when passed

19. Copying images Be careful to remember that most image copy and pass by value methods do NOT perform a deep copy. image2 = image1; // shallow copy void someMethod(Mat imageParam); // shallow copy If you want a deep copy, then use clone (or copyTo): image2 = image1.clone(); // deep copy image1.copyTo(image2); // deep copy

20. Memory management Memory management is handled by the Mat class. • This is different from the IplImage class in OpenCV 1 • This works correctly even if multiple images share the same data • A reference count is kept for each image • The data is deallocated only when the reference count goes to zero • However, this can allow privacy leaks unless you are careful

21. Backwards compatibility OpenCV 2 is backwards compatible with OpenCV 1. IplImage*iplIm = cvLoadImage("photo.jpg"); // Do some work with image here cvReleaseImage(&iplIm); // necessary to prevent memory leak Can convert to Mat simply: Mat converted(iplIm); // do not release image until finished

22. Image manipulation OpenCV provides many methods to manipulate entire images: • Filtering: blur, smooth, median, gradient, laplacian • Transformations: resize, affine, perspective, color space, threshold, flood fill • Segmentation: grabCut, watershed • Feature detection: edges, corners, lines, circles, template matching, SIFT