Local invariant features

1. Local invariant features Cordelia Schmid INRIA, Grenoble

2. ( ) Local features local descriptor Several / many local descriptors per image Robust to occlusion/clutter + no object segmentation required Photometric : distinctive Invariant : to image transformations + illumination changes

3. Local features: interest points

4. Local features: Contours/segments

5. Local features: segmentation

6. Application: Matching Find corresponding locations in the image

7. Matching algorithm • Extract descriptors for each image I1 and I2 • Compute similarity measure between all pairs of descriptors • Select couples according to different strategies • All matches above a threshold • Winner takes all • Cross-validation matching • Verify neighborhood constraints • Compute the global geometric relation (fundamental matrix or homography) robustly 6. Repeat matching using the global geometric relation

8. Selection strategies • Winner takes all • The best matching pairs (with the highest score) is selected • All matches with the points and are removed • Cross-validation matching • For each point in image 1 keep the best match • For each point in image 2 keep the best match • Verify the matches correspond both ways

9. Illustration – Matching Interest points extracted with Harris detector (~ 500 points)

10. Illustration – Matching Matching Interest points matched based on cross-correlation (188 pairs)

11. Illustration – Matching Global constraints Global constraint - Robust estimation of the fundamental matrix 89 outliers 99 inliers

12. Application: Panorama stitching Images courtesy of A. Zisserman.

13. Application: Instance-level recognition Search for particular objects and scenes in large databases …

14. Difficulties Finding the object despite possibly large changes in scale, viewpoint, lighting and partial occlusion  requires invariant description Scale Viewpoint Occlusion Lighting

15. Difficulties • Very large images collection  need for efficient indexing • Flickr has 2 billion photographs, more than 1 million added daily • Facebook has 15 billion images (~27 million added daily) • Large personal collections • Video collections, i.e., YouTube

16. Applications Search photos on the web for particular places ...in these images and 1M more Find these landmarks

17. Applications • Take a picture of a product or advertisement  find relevant information on the web [Pixee – Milpix]

18. Applications • Finding stolen/missing objects in a large collection …

19. Applications • Copy detection for images and videos Search in 200h of video Query video

20. Sony Aibo – Robotics Recognize docking station Communicate with visual cards Place recognition Loop closure in SLAM Applications 20 K. Grauman, B. Leibe Slide credit: David Lowe

21. Instance-level recognition: Approach • Extraction of invariant image descriptors • Matching descriptors between images • Matching of the query images to all images of a database • Speed-up by efficient indexing structures • Geometric verification • Verification of spatial consistency for a short list

22. Local features - history • Line segments [Lowe’87, Ayache’90] • Interest points & cross correlation [Z. Zhang et al. 95] • Rotation invariance with differential invariants [Schmid&Mohr’96] • Scale & affine invariant detectors [Lindeberg’98, Lowe’99, Tuytelaars&VanGool’00, Mikolajczyk&Schmid’02, Matas et al.’02] • Dense detectors and descriptors [Leung&Malik’99, Fei-Fei& Perona’05, Lazebnik et al.’06] • Contour and region (segmentation) descriptors [Shotton et al.’05, Opelt et al.’06, Ferrari et al.’06,Leordeanu et al.’07]

23. Local features 1) Extraction of local features • Contours/segments • Interest points & regions • Regions by segmentation • Dense features, points on a regular grid 2) Description of local features • Dependant on the feature type • Segments  angles, length ratios • Interest points  greylevels, gradient histograms • Regions (segmentation)  texture + color distributions

24. Line matching • Extraction de contours • Zero crossing of Laplacian • Local maxima of gradients • Chain contour points (hysteresis) • Extraction of line segments • Description of segments • Mi-point, length, orientation, angle between pairs etc.

25. Experimental results – line segments images 600 x 600

26. Experimental results – line segments 248 / 212 line segments extracted

27. Experimental results – line segments 89 matched line segments - 100% correct

28. Experimental results – line segments 3D reconstruction

29. Problems of line segments • Often only partial extraction • Line segments broken into parts • Missing parts • Information not very discriminative • 1D information • Similar for many segments • Potential solutions • Pairs and triplets of segments • Interest points

30. Overview • Harris interest points • Comparison of IP: SSD, ZNCC, SIFT • Scale & affine invariant interest points (student presentation) • Evaluation and comparison of different detectors • Region descriptors and their performance

31. Harris detector [Harris & Stephens’88] Based on the idea of auto-correlation Important difference in all directions => interest point

32. Harris detector Auto-correlation function for a point and a shift

33. Harris detector Auto-correlation function for a point and a shift { → uniform region small in all directions → contour large in one directions → interest point large in all directions

34. Harris detector

35. with first order approximation Harris detector Discret shifts are avoided based on the auto-correlation matrix

36. the sum can be smoothed with a Gaussian Harris detector Auto-correlation matrix

37. Harris detector • Auto-correlation matrix • captures the structure of the local neighborhood • measure based on eigenvalues of this matrix • 2 strong eigenvalues • 1 strong eigenvalue • 0 eigenvalue => interest point => contour => uniform region

38. Interpreting the eigenvalues Classification of image points using eigenvalues of M: 2 “Edge” 2 >> 1 “Corner”1 and 2 are large,1 ~ 2;E increases in all directions 1 and 2 are small;E is almost constant in all directions “Edge” 1 >> 2 “Flat” region 1

39. Corner response function α: constant (0.04 to 0.06) “Edge” R < 0 “Corner”R > 0 |R| small “Edge” R < 0 “Flat” region

40. Reduces the effect of a strong contour Harris detector • Interest point detection • Treshold (absolut, relatif, number of corners) • Local maxima • Cornerness function

41. Harris Detector: Steps

42. Harris Detector: Steps Compute corner response R

43. Harris Detector: Steps Find points with large corner response: R>threshold

44. Harris Detector: Steps Take only the points of local maxima of R

45. Harris Detector: Steps

46. Harris detector: Summary of steps • Compute Gaussian derivatives at each pixel • Compute second moment matrix M in a Gaussian window around each pixel • Compute corner response function R • Threshold R • Find local maxima of response function (non-maximum suppression)

47. Harris - invariance to transformations • Geometric transformations • translation • rotation • similitude (rotation + scale change) • affine (valide for local planar objects) • Photometric transformations • Affine intensity changes (I  a I + b)

48. Harris Detector: Invariance Properties • Rotation Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation

49. Intensity scale: I  aI R R threshold x(image coordinate) x(image coordinate) Harris Detector: Invariance Properties • Affine intensity change • Only derivatives are used => invariance to intensity shift I I+b Partially invariant to affine intensity change, dependent on type of threshold

50. Harris Detector: Invariance Properties • Scaling Corner All points will be classified as edges Not invariant to scaling