Feature Detection

1. Feature Detection Giacomo Boracchi 6/12/2007 boracchi@elet.polimi.it

2. Feature matching vs. tracking • Image-to-image correspondences are determined by some salient point. • These are also the basis of passive triangulation-based 3D reconstruction • Feature Matching: Extract features independently and then match by comparing descriptors • Feature Tracking: Extract features in first images and then try to find same feature back in next view What is a good feature?

3. Feature Properties • Well-defined: i.e. neighboring points should all be different • Stable across views: same scene point should be extracted as feature for neighboring viewpoints

4. Interest Point Detection Low Level Inspired • Gradient Based (ex Harris, Hessian) • Phase Based (Kovesi) • Entropy Based (Zisserman)

5. Comparing image regions: Pixel by Pixel • Dissimilarity Measure SSD the Sum of Squared Distances I´(x,y) I(x,y)

6. Harris – Moravec • Compute the Sum of Square Distances between the image values on the green square at different position “flat” region:no change in all directions “edge”:no change along the edge direction “corner”:significant change in all directions

7. 2 ( ) P ( ) ( ( ) ( ) ) E I I ¡ ¡ ¡ x y w u v u v u x v y = ; ; ; ; u v Moreavec (80) • Response on sliding windows – Moravec (81) • w represent the window (green in the previous slides) • Corner, even small changes are “big” • where is a threshold • Look for local maxima in min{E} above some threshold

8. µ ¶ 2 ( ( ) ) P ( ) ( ( ) ( ) ) r E E I I 0 0 0 x ¡ ¡ ¡ x y w u v u v u x v y ¡ ¢ ¡ ¢ 1 = = ( ) ( ) E E r E H 0 0 + + ; ; ; ; ; ¼ u x y x y x y v ; ; 2 y Moravec Drawbacks – Solutions • The response is isotropic as only a finite set of displacements (x,y) is considerer • therefore, the same corner rotated may yield different responses. • Solution: Analytical Formulation of E(x,y), expand it in Taylor series • Now E(0,0) = 0 and one can prove that also the term vanishes • H denotes the Hessian Matrix but when it is expressed as a function of I we call it M

9. µ ¶ x ¡ ¢ ( ) E M ¼ x y x y ; y Moravec Drawbacks – Solutions • Now we can approximatebeing the derivative computed with Sobel or Previtt filters

10. 2 “Edge” 2 >> 1 “Corner”1 and 2 are large,1 ~ 2;E increases in all directions “Flat” region “Edge” 1 >> 2 1 Moravec Drawbacks – Solutions • Considering only the minimum of E is not a great deal, may give too ready responses • Solution consider the SVDof M and pretend that theminimum of the eigenvalues of M is big

11. Moravec Drawbacks – Solutions • The response may be noisy • Solution: take w as Gaussian distributed Weights

12. Harris – Stevens (88) • To avoid eigenvalue decomposition 2 “Edge” 2 >> 1 “Corner”1 and 2 are large,1 ~ 2;E increases in all directions “Flat” region “Edge” 1 >> 2 1 Alternatively

13. Feature point extraction Find points that differ as much as possible from all neighboring points homogeneous edge corner

14. Feature point extraction homogeneous edge corner Find points for which the following is maximum i.e. maximize smallest eigenvalue of M

15. Harris corner detector as Feature Selector • Select the local maxima of Harris Measure as Features to perform Matching • Only use local maxima, subpixel accuracy through second order surface fitting

16. Harris – Stevens (88) : Feature extraction • Pro • Fast • Rotation invariant • Shortcomings • No scale invariant • No affine transform invariant

17. FeatureTracking Giacomo Boracchi 6/12/ 2007

18. Comparing image regions: Pixel by Pixel • Dissimilarity Measure SSD the Sum of Squared Distances I´(x,y) I(x,y)

19. Comparing image regions: Pixel by Pixel • Similarity measures I´(x,y) I(x,y)

20. Normalized Cross Correlation • Gives a measure between [-1,1] • While the SSD is only positive and not normalized • Independent on Image Intensity • Allows fast implementation using running sum to compute local averages

21. Tracking Problems • Occlusions • Feature Deformations due to motions • Affine • Perspective • Light Changes (not for NCC based measures) • Noise • Blur • Other Artifacts (e.g. Jpeg compression)

22. ( ( ) ) ( ( ) ) ( ) » » d » l ± » I I t t t + + + x y x y ¿ x a n ´ y ´ ´ x y ¿ ¿ e ´ = = = = ; ; ; ; ; ; ; ; ; KLT – Shi and Tomasi 94 • Take into account also for affine changes in the image • Consider a Video I(x,y,t) it’s the value of the pixel x,y in the image at time t • Then how to relate the same intensity in two different images • And extending such requirement to a neighbor we get feature matching • And, of course

23. [ ] d d d = x y ; KLT – Shi and Tomasi 94: Feature Tracking • Being the displacement represented by an affine motion field • Consider then only two frames I,J • Affine motion model • Translation only

24. KLT – Shi and Tomasi 94: Then what Tracking actually is • Given two images I and J we are interested in determiningthe six parameters (4 affine + 2 translational motion) • Then for consequent frame a pure translation model is commonly assumed (i.e. D= Identity)

25. d d f f k k b b d d T Z T Z 6 2 t t t t z a e z v v e e c c o o r r o o u u n n n n o o w w n n s s e a c c a a n n e e c c o o m m p p u u e e = = , , , , KLT – Shi and Tomasi 94 : Dissimilarity measure • Goal : minimize dissimilarity (SSD based tracking) • A and d can be determined in a closed form (approximate solutions) • An exact solution can be obtained by iterative procedure • And also for the translation only case

26. KLT – Shi and Tomasi 94: Which are Good features? • ”The right features are exactly those that make the tracker work best” • How the tracker work? • Inverting 2 matrices Z and T • Then Good features are those having the matrices Z and T well conditioned (then again eigenvalues analysis) • Thus, no a priori assumption about feature goodness • Affine model also considered

27. KLT – Shi and Tomasi 94 : Performances • Adjacent frames – pure displacement Z matrix • Cumulative changes – Affine Transform T matrix

28. KLT – Shi and Tomasi 94 : Performances • Camera moving forward 2mm per frame • Affine alignment between first and last frame • Stop tracking features with too large errors

29. KLT – Shi and Tomasi 94 : Performances • Dissimilarity computed using pure displacement. • Dissimilarity computed using affine transforms

30. A Project: Rotational Blur Removal Giacomo Boracchi

31. Rotational Blur • In each pixel the blur smears are varying • Only in trivial case of rotation axis orthogonal to image plane this can be considered circumferences

32. Rotation Axis Estimation: the local blur analysis

33. Rotation Axis Estimation: the voting procedure

34. 2 ¼ P C ´ ¼ ¼ Blurring paths • blurring path the set of pixels which are intersected by a viewing ray after a rotation of around the axis a C a V P V a circumferences Conic sections

35. Blur Removal: Scheme and Issues • Projection of the image on the plane orthogonal to rotation axis • In such a way the blurring paths become circumferences • The mapping (an homography) should be extremely accurate • Cartesian to Polar coordinate transform so that the blur becomes shift invariant, i.e. uniform, all the pixels blurred at the same way • Blur Inversion via Regularized Wiener Filtering (Classical Procedure) • Polar to Cartesian coordinate transform • Denoising in Cartesian domain in order to exploit image structures • Using LASIP adaptive filtering algorithms (codes) • Noise Modeling via Monte Carlo approach

36. License Plate Recognition • Objective: automatically detect and recognize license plate from video sequences on a dsp-equipped camera. YZH 4025

37. License Plate Recognition – Available Projects • New Starting Project • Probably Strict Deadlines • ONLY THE BRAVES! • License Plate Detection Module • Develop a module that detect the license plate in image according to color and shape • License Plate Recognition Module • Given the license plate image, develop a module that recognize characters (e.g. using a neural network) and provide the license number • Both projects require good C-programming skills • Coordinators: Caglioti, Gasparini, Taddei

38. Computer Vision Group Team • Vincenzo Caglioti • Giacomo Boracchi, Simone Gasparini, Alessandro Giusti, Pierluigi Taddei