OpenCV Tutorial

1. OpenCV Tutorial Omri Perez Adapted from: Gary Bradski Senior Scientist, Willow Garage Consulting Professor: Stanford CS Dept. http://opencv.willowgarage.com www.willowgarage.com 1

2. Vision is Hard • Camera Model, Lens, Problems and Corrections • OpenCV • OpenCV Tour CS324

3. Vision is Hard • What is it? • Turning sensor readings into perception. • Why is it hard? • It’s just numbers. Maybe try gradients to find edges? CS324

4. Use Edges? … It’s not so simple • Depth discontinuity • Surface orientation discontinuity • Reflectance discontinuity (i.e., change in surface material properties) • Illumination discontinuity (e.g., shadow) Slide credit: Christopher Rasmussen CS324

5. Must deal with Lighting Changes … CS324

6. Lighting is also a Strong Cue Gary Bradski (c) 2008 6

7. The Brain Assumes 3D Geometry Perception is ambiguous … depending on your point of view! 7

8. Geometrical aberrations Non-Geometrical aberrations • spherical distortion • astigmatism • tangential distortion • coma • Chromatic • Vignetting These are typically what are corrected for in camera Calibration aberrations are reduced by combining lenses Marc Pollefeys

9. Distortion Correction so that Lenscan Approximate a Pinhole Camera • Distortions are corrected mathematically • We use a calibration pattern • We find where the points ended up • We know where the points hould be • OpenCV 2.2 Function: double calibrateCamera( const vector<vector<Point3f> >& objectPoints, const vector<vector<Point2f> >& imagePoints, Size imageSize, Mat& cameraMatrix, Mat& distCoeffs, vector<Mat>& rvecs, vector<Mat>& tvecs, int flags=0); CS324

10. Vision is Hard • Camera Model, Lens, Problems and Corrections • OpenCV • OpenCV Tour CS324

11. OpenCV Overview: Robot support > 2000 algorithms opencv.willowgarage.com Image Pyramids General Image Processing Functions Geometric descriptors Camera calibration, Stereo, 3D Segmentation Features Utilities and Data Structures Transforms Tracking Machine Learning: • Detection, • Recognition Fitting Matrix Math Gary Bradski

12. OpenCV Tends Towards Real Time http://opencv.willowgarage.com

13. Where is OpenCV Used? • Google Maps, Google street view, Google Earth, Books • Academic and Industry Research • Safety monitoring (Dam sites, mines, swimming pools) • Security systems • Image retrieval • Video search • Structure from motion in movies • Machine vision factory production inspection systems • Robotics • Well over 2M downloads 2M downloads Screen shots by Gary Bradski, 2005

14. OpenCV Modules • Calib3d • Calibration, stereo, homography, rectify, projection, solvePNP • Contrib • Octree, self-similar feature, sparse L-M, bundle adj, chamfer match • Core • Data structures, access, matrix ops, basic image operations • features2D • Feature detectors, descriptors and matchers in one architecture • Flann (Fast library for approximate nearest neighbors) • Gpu – CUDA speedups • Highgui • Gui to read, write, draw, print and interact with images • Imgproc – image processing functions • Ml – statistical machine learning, boosting, clustering • Objdetect – PASCAL VOC latent SVM and data reading • Traincascade – boosted rejection cascade CS324

15. Software Engineering • Works on: • Linux, Windows, Mac OS (+ Android since open CV 2.2) • Languages: • C++, Python, C • Online documentation: • Online reference manuals: C++, C and Python.

16. Vision is Hard • Camera Model, Lens, Problems and Corrections • OpenCV • OpenCV Tour CS324

17. Gradients: Scharr instead of Sobel • Sobel has been the traditional 3x3 gradient finder. • Use the 3x3 Scharr operator instead since it is just as fast but has more accurate response on diagonals. void Scharr(const Mat& src, Mat& dst, int ddepth, int xorder, int yorder, double scale=1, double delta=0, int borderType=BORDER_DEFAULT) CS324

18. Canny Edge Detector Canny() OpenCV team, Gary Bradski 18

19. Hough Transform HoughCircles(), HoughLines(), HoughLinesP() (probabilistic Hough) Gary Bradski (c) 2008 19

20. Scale Space void cvPyrUp( IplImage* src, IplImage* dst, IplFilter filter = IPL_GAUSSIAN_5x5); void cvPyrDown( IplImage* src, IplImage* dst, IplFilter filter = IPL_GAUSSIAN_5x5); Gary Bradski (c) 2008 20

21. Space Variant vision: Log-Polar Transform cvLogPolar(src,dst,center,size, CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS) Gary Bradski (c) 2008 21

22. Delaunay Triangulation, Voronoi Tessellation CvSubdiv2D* cvCreateSubdivDelaunay2D(CvRect rect, CvMemStorage* storage) Gary Bradski (c) 2008 22

23. Contours void findContours() Gary Bradski (c) 2008 23

24. Histogram Equalization void equalizeHist(const Mat& src, Mat& dst) Gary Bradski (c) 2008 24

25. Image textures • Inpainting: • Removes damage to images, in this case, it removes the text. void inpaint(const Mat& src, const Mat& inpaintMask, Mat& dst, double inpaintRadius, int flags); Gary Bradski (c) 2008 25

26. Morphological Operations Examples • Morphology - applying Min-Max. Filters and its combinations Void morphologyEx() createMorphologyFilter() erode() dilate() Dilatation IB Image I Erosion IB Opening IoB= (IB)B Grad(I)= (IB)-(IB)‏ Closing I•B= (IB)B TopHat(I)= I - (IB)‏ BlackHat(I)= (IB) - I Gary Bradski (c) 2008 26

27. Distance Transform • Distance field from edges of objects void distanceTransform(const Mat& src, Mat& dst, int distanceType, int maskSize) Flood Filling int floodFill(Mat& image, Point seed, Scalar newVal, Rect* rect=0, Scalar loDiff=Scalar(), Scalar upDiff=Scalar(), int flags=4) 27 Gary Bradski (c) 2008

28. Thresholds void adaptiveThreshold() double threshold() Gary Bradski (c) 2008 28

29. Segmentation • Pyramid, mean-shift, graph-cut • Here: Watershed void watershed(const Mat& image, Mat& markers) Gary Bradski (c) 2008 29

30. Background Subtraction BackgroundSubtractorMOG2(), see samples/cpp/bgfg_segm.cpp Gary Bradski (c) 2008 30

31. Image Segmentation & Minimum Cut Pixel Neighborhood w Image Pixels Similarity Measure Minimum Cut * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

32. GrabCut void grabCut(const Mat& image, Mat& mask, Rect rect, Mat& bgdModel, Mat& fgdModel, int iterCount, int mode) • Graph Cut based segmentation Gary Bradski

33. Motion Templates (My work with James Davies)‏ • Object silhouette • Motion history images • Motion history gradients • Motion segmentation algorithm silhouette MHI MHG Gary Bradski (c) 2008 33

34. Segmentation, Motion Tracking void updateMotionHistory(); void calcMotionGradient(); double calcGlobalOrientation(); Motion Segmentation Motion Segmentation Pose Recognition Gesture Recognition 34 Gary Bradski (c) 2008 James Davies, Gary Bradski

35. Tracking with CAMSHIFT • Control game with head RotatedRect CamShift(const Mat& probImage, Rect& window, TermCriteria criteria)

36. 3D tracking • Camera Calibration • View Morphing • POSIT • void POSIT() • A more general technique for solving pose is • solving the Percpective N Point problem: • void solvePnP(…)

37. Mean-Shift for Tracking CamShift(); MeanShift(); Gary Bradski (c) 2008 37

38. Optical Flow // opencv/samples/c/lkdemo.c int main(…){ … CvCapture* capture = <…> ? cvCaptureFromCAM(camera_id) : cvCaptureFromFile(path); if( !capture ) return -1; for(;;) { IplImage* frame=cvQueryFrame(capture); if(!frame) break; // … copy and process image cvCalcOpticalFlowPyrLK( …)‏ cvShowImage( “LkDemo”, result ); c=cvWaitKey(30); // run at ~20-30fps speed if(c >= 0) { // process key }} cvReleaseCapture(&capture);} calcOpticalFlowPyrLK() Also see dense optical flow: calcOpticalFlowFarneback()

39. Features 2D Read two input images: Mat img1 = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE); Detect keypoints in both images: // detecting keypoints FastFeatureDetector detector(15); vector<KeyPoint> keypoints1; detector.detect(img1, keypoints1); Compute descriptors for each of the keypoints: // computing descriptors SurfDescriptorExtractor extractor; Mat descriptors1; extractor.compute(img1, keypoints1, descriptors1); Now, find the closest matches between descriptors from the first image to the second: // matching descriptors BruteForceMatcher<L2<float> > matcher; vector<DMatch> matches; matcher.match(descriptors1, descriptors2, matches); CS324

40. Features 2D continued … Viusalize the results namedWindow("matches", 1); Mat img_matches; drawMatches(img1, keypoints1, img2, keypoints2, matches, img_matches); imshow("matches", img_matches); waitKey(0); Find the homography transformation between two sets of points: vector<Point2f> points1, points2; // fill the arrays with the points .... Mat H = findHomography(Mat(points1), Mat(points2), CV_RANSAC, ransacReprojThreshold); Create a set of inlier matches and draw them. Use perspectiveTransform function to map points with homography: Mat points1Projected; perspectiveTransform(Mat(points1), points1Projected, H); Use drawMatches() again for drawing inliers. CS324

41. Features2d contents Detection: Description: Detectors available Descriptors available SIFT SURF One way Calonder (under construction) FERNS • SIFT • SURF • FAST • STAR • MSER • GFTT (Good Features To Track)

42. Kalman Filter, Partical Filter for Tracking Kalman Condensation or Particle Filter ::KalmanFilter class ConDensation 42 Gary Bradski (c) 2008

43. Projections Find: Warp: Mat getAffineTransform() void warpAffine() Mat getPerspectiveTransform() void warpPerspective()

44. Homography • Maps one plane to another • In our case: A plane in the world to the camera plane • Great notes on this: Robert Collins CSE486 • http://www.cse.psu.edu/~rcollins/CSE486/lecture16.pdf • Derivation details: Learning OpenCV 384-387 Perspective Matrix Equation (camera coords Pt in world to pt on image) Gary Bradski and Adrian Kaehler: Learning OpenCV Gary Bradski, CS223A, Into to Robotics

45. Homography • We often use the chessboard detector to find 4 non-colinear points • (X,Y * 4 = 8 constraints) • To solve for the 8 homography parmeters. • Code: Once again, OpenCV makes this easy • findHomography(…) or: • getPerspectiveTransform(…) Gary Bradski, CS223A, Into to Robotics

46. Single Camera Calibration See samples/cpp/calibration.cpp Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds. And after that you’ll get: 3D view of checkerboard Un-distorted image Gary Bradski (c) 2008 46

47. Stereo … Depth from Triangulation • Involved topic, here we will just skim the basic geometry. • Imagine two perfectly aligned image planes: Depth “Z” and disparity “d” are inversly related:

48. Stereo • In aligned stereo, depth is from similar triangles: • Problem: Cameras are almost impossible to align • Solution: Mathematically align them: All: Gary Bradski and Adrian Kaehler: Learning OpenCV

49. Stereo Rectification • Algorithm steps are shown at right: • Goal: • Each row of the image contains the same world points • “Epipolar constraint” Result: Epipolar alignment of features: All: Gary Bradski and Adrian Kaehler: Learning OpenCV

50. samples/c In ...\opencv_incomp\samples\c bgfg_codebook.cpp - Use of a image value codebook for background detection for collecting objects bgfg_segm.cpp - Use of a background learning engine blobtrack.cpp - Engine for blob tracking in images calibration.cpp - Camera Calibration camshiftdemo.c - Use of meanshift in simple color tracking contours.c - Demonstrates how to compute and use object contours convert_cascade.c - Change the window size in a recognition cascade convexhull.c - Find the convex hull of an object delaunay.c - Triangulate a 2D point cloud demhist.c - Show how to use histograms for recognition dft.c - Discrete fourier transform distrans.c - distance map from edges in an image drawing.c - Various drawing functions edge.c - Edge detection facedetect.c - Face detection by classifier cascade ffilldemo.c - Flood filling demo find_obj.cpp - Demo use of SURF features fitellipse.c - Robust elipse fitting houghlines.c - Line detection image.cpp - Shows use of new image class, CvImage(); inpaint.cpp - Texture infill to repair imagery kalman.c - Kalman filter for trackign kmeans.c - K-Means laplace.c - Convolve image with laplacian. letter_recog.cpp - Example of using machine learning Boosting, Backpropagation (MLP) and Random forests lkdemo.c - Lukas-Canada optical flow minarea.c - For a cloud of points in 2D, find min bounding box and circle. Shows use of Cv_SEQ morphology.c - Demonstrates Erode, Dilate, Open, Close motempl.c - Demonstrates motion templates (orthogonal optical flow given silhouettes) mushroom.cpp - Demonstrates use of decision trees (CART) for recognition pyramid_segmentation.c - Color segmentation in pyramid squares.c - Uses contour processing to find squares in an image stereo_calib.cpp - Stereo calibration, recognition and disparity map computation watershed.cpp - Watershed transform demo.