OpenCV Introduction

1. OpenCV Introduction Hang Xiao Oct 26, 2012

2. History • 1999 Jan : lanchedby Intel, real time machine vision library for UI, optimized code for intel • 2000Jun : OpenCValpha 3。 • 2000 Dec : OpenCVbeta 1for linux • 2006 : the first 1.0 version supports Mac OS • 2008 mid : obtain corporate support from Willow Garage • 2009Sep : OpenCV1.2（beta2.0 • 2009Oct: Version 2.0 released。 • 2010Dec: OpenCV 2.2。 • 2011Aug : OpenCV 2.3。 • 2012Apr : OpenCV2.4.

3. Overview • Goals • Develop a universal toolbox for research and development in the field of Computer Vision • Algorithms • More than 350algorithms, 500API • Programming language • C/C++, C#, Ch , Python, Ruby, Matlab, and Java (using JavaCV) • OS support • Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac OS. • Licence • BSDlisence, free for commercial and non-commmercial

4. Overview - Applications • 2D and 3D feature toolkits • Egomotion estimation • Facial recognition system • Gesture recognition • Human–computer interaction (HCI) • Mobile robotics • Motion understanding • Object identification • Segmentation and Recognition • Stereopsis Stereo vision: depth perception from 2 cameras • Structure from motion (SFM)Motion tracking

5. Overview - A statistical machine learning library • Boosting (meta-algorithm) • Decision tree learning • Gradient boosting trees • Expectation-maximization algorithm • k-nearest neighbor algorithm • Naive Bayes classifier • Artificial neural networks • Random forest • Support vector machine (SVM)

6. Outline • Image Analysis • Structural Analysis • Object Recognition • Motion Analysis and Object Tracking • 3D Reconstruction

7. Outline • Image Analysis • Structural Analysis • Object Recognition • Motion Analysis and Object Tracking • 3D Reconstruction

8. Image Analysis • Thresholds • Statistics • Pyramids • Morphology • Distance transform • Flood fill • Feature detection • Contours retrieving

9. Image Thresholding • Fixed threshold; • Adaptive threshold;

10. Image Thresholding Examples Source picture Fixed threshold Adaptive threshold

11. Statistics • min, max, mean value, standard deviation over the image • Norms C, L1, L2 • Multidimensional histograms • Spatial moments up to order 3 (central, normalized, Hu)

12. Multidimensional Histograms • Histogram operations • calculation, normalization, comparison, back project • Histograms types: • Dense histograms • Signatures (balanced tree) • EMD algorithm • The EMD computes the distance between two distributions, which are represented by signatures. • The signatures are sets of weighted features that capture the distributions. The features can be of any type and in any number of dimensions, and are defined by the user. • The EMD is defined as the minimum amount of work needed to change one signature into the other

13. EMD – a method for the histograms comparison

14. Image Pyramids • Gaussian and Laplacian pyramids • Image segmentation by pyramids

15. Image Pyramids • Gaussian and Laplacian

16. Pyramid-based color segmentation On still pictures And on movies

17. Morphological Operations • Two basic morphology operations using structuring element: • erosion • dilation • More complex morphology operations: • opening : erosion + dilation • closing : dilation + erosion • morphological gradient : the difference between the dilation and the erosion of an image • top hat : the difference between an input image and its opening • black hat : the difference between the closing and its input image

18. Morphological Operations Examples • Morphology - applying Min-Max. Filters and its combinations Dilatation IB Image I Erosion IB Opening IoB= (IB)B Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB) - I Grad(I)= (IB)-(IB)

19. Distance Transform • Calculate the distance for all non-feature points to the closest feature point • Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined

20. Flood Filling • Simple • Gradient

21. Feature Detection • Fixed filters (Sobel operator, Laplacian); • Optimal filter kernels with floating point coefficients (first, second derivatives, Laplacian) • Special feature detection (corners) • Canny operator • Hough transform (find lines and line segments) • Gradient runs

22. Canny Edge Detector

23. Detects lines in a binary image Hough Transform • Probabilistic Hough Transform • Standard Hough Transform

24. Another Sample of the Hough Transform Using Source picture Result

25. Contour Retrieving • The contour representation: • Chain code (Freeman code) • Polygonal representation Initial Point Chain code for the curve: 34445670007654443 Contour representation

26. Hierarchical representation of contours Image Boundary (W1) (W2) (W3) (B2) (B3) (B4) (W5) (W6)

27. Contours Examples Source Picture (300x600 = 180000 pts total) Retrieved Contours (<1800 pts total) After Approximation (<180 pts total) And it is rather fast: ~70 FPS for 640x480 on complex scenes

28. Outline • Image Analysis • Structural Analysis • Object Recognition • Motion Analysis and Object Tracking • 3D Reconstruction

29. Structural Analysis • Contours processing • Approximation • Hierarchical representation • Shape characteristics • Matching • Geometry • Contour properties • Fitting with primitives • PGH:pair-wise geometrical histogram for the contour.

30. Contour Processing • Approximation: • RLE algorithm (chain code) • Teh-Chin approximation (polygonal) • Douglas-Peucker approximation (polygonal); • Contour moments (central and normalized up to order 3) • Hierarchical representation of contours • Matching of contours

31. Hierarchical Representation of Contours • A contour is represented with a binary tree • Given the binary tree, the contour can be retrieved with arbitrary precision • The binary tree is quasi invariant to translations, rotations and scaling

32. Contours matching • Matching based on hierarchical representation of contours

33. Geometry • Properties of contours: (perimeter, area, convex hull, convexity defects, rectangle of minimum area) • Fitting: (2D line, 3D line, circle, ellipse) • Pair-wise geometrical histogram

34. Pair-wise geometrical histogram (PGH)

35. Outline • Image Analysis • Structural Analysis • Object Recognition • Motion Analysis and Object Tracking • 3D Reconstruction

36. Object Recognition • Eigen objects • Hidden Markov Models

37. Eigen Objects

38. Eigen objects (continued)

39. Hidden Markov ModelDefinitions - The set of states - The set of measurements - The state at timet - The transition probability matrix - The conditional probability matrix - The starting states distribution

40. Embedded HMM for Face Recognition Model- - Face ROI partition

41. Face recognition using Hidden Markov Models • One person – one HMM • Stage 1 – Train every HMM • Stage 2 – Recognition Pi - probability Choose max(Pi) 1 … n i

42. Outline • Image Analysis • Structural Analysis • Object Recognition • Motion Analysis and Object Tracking • 3D Reconstruction

43. Motion Analysis and Object Tracking • Background subtraction • Motion templates • Optical flow • Active contours • Estimators

44. Background Subtraction • Background model (normal distribution) • Background statistics functions: • Average • Standard deviation • Running average

45. Motion Templates • Object silhouette • Motion history images • Motion history gradients • Motion segmentation algorithm MHG silhouette MHI

46. Motion Segmentation Algorithm • Two-pass algorithm labeling all motion segments

47. Motion Templates Example • Motiontemplatesallow to retrieve the dynamic characteristics of the moving object

48. Optical Flow • Block matching technique • Horn & Schunck technique • Lucas & Kanade technique • Pyramidal LK algorithm • 6DOF (6 degree of freedom) algorithm Optical flow equations:

49. Pyramidal Implementation of the optical flow algorithm Image Pyramid Representation Iterative Lucas – Kanade Scheme J image I image Location of point u on image uL=u/2L Spatial gradient matrix Standard Lucas – Kanade scheme for optical flow computation at level L dL Guess for next pyramid level L – 1 Finally, Generic Image (L-1)-th Level Image pyramid building L-th Level Optical flow computation

50. 6DOF Algorithm Parametrical optical flow equations: