SIFT Scale Invariant Feature Transform

1. SIFTScale Invariant Feature Transform Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer Vision, 60, 2 (2004), pp. 91-110 Presented by:Shalomi Eldar Based (in part) on slides by Ofir Pele, Kirill Dyagilev and Ayelet Dominitz Vision Topics Seminar 2009

2. Description Applications Introduction Detection Image Matching Fundamental aspect of many problems: Object Recognition 3D structures Stereo Correspondence Motion Tracking

3. Description Applications Introduction Detection Features Detection • Give comprehensive description of image. • Enables matching! What are the desired features’ features? Images from: M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the )the International Conference on Computer Vision (ICCV2003

4. Features’ Features Description Applications Introduction Detection • Robustness => Invariance to changes in illumination, scale, rotation, affine, perspective. • Locality => robustness to occlusion and clutter. • Distinctiveness => easy to match to a large database of objects. • Quantity => many features can be generated for even small objects. • Efficiency => computationally “cheap”, real-time performance.

5. SIFT Algorithm Description Applications Introduction Detection • Input: • Image nxm. • Output: • Set of descriptors of image’s features. SIFT - Scale Invariant Feature Transform

6. SIFT Algorithm Description Applications Introduction Detection • Scale-space extrema detection. • Keypoint localization. Typical image of size 500x500 pixels produces about 2000 stable keypoints • Orientation assignment. • Keypoint descriptor. • Near Real-time performance: Cascade approach – keep heavy operations only to keypoints that “survive”.

7. SIFT Algorithm Description Applications Introduction Detection • Scale-space extrema detection. • Keypoint localization. • Orientation assignment. • Keypoint descriptor. - matching. => Application We need only 3 keypoints matches for reliable identification!

8. Description Applications Introduction Detection Today’s Talk Orientation Z Z Z Z Z Detection Description Application Extracting Keypoints Distinctive Description Matching Keypoints • Extrema detection • Correct localization • Choosing robust keypoints only • Local invariant orientation • Building keypoint descriptor • Nearest-neighbor algorithm • Finding 3 matches • Least-square affine approximation 1st part of the talk 2nd part of the talk Last part of the talk

9. SIFT Algorithm Description Applications Introduction Detection • Scale-space extrema detection. • Keypoint localization. Collecting keypoint candidates… • Orientation assignment. • Keypoint descriptor.

10. Description Applications Introduction Detection Why Extrema? • We want to find points that give us information about the objects in the image. • The information about the objects is in the object’s edges. • We will represent the image in a way that gives us these edges as this representations extrema points.

11. Description Applications Introduction Detection Scale-space Representation

12. Description Applications Introduction Detection Scale-space Representation • Difference-of-Gaussian (DoG): Low computation time - Only subtraction of smoothed images!

13. Description Applications Introduction Detection DoG Pyramid • Here’s what we get: Scale invariance Different frequencies features

14. Description Applications Introduction Detection Extracting Keypoints • X is selected if it is larger or smaller than all 26 neighbors. Low cost - only several usually checked.

15. Description Applications Introduction Detection • Each Keypoint is represented as . Extracting Keypoints • Extrema detection product: 233x189 image => 832 DoG Keypoints. Not all of them are good…

16. Description Applications Introduction Detection Problematic Keypoints • Inaccurate localization (due to scaling and sampling). • Low contrast - sensitive to noise. • Strong edge responses.

17. SIFT Algorithm Description Applications Introduction Detection • Scale-space extrema detection. • Keypoint localization. • Orientation assignment. Filtering Keypoints… • Keypoint descriptor.

18. Description Applications Introduction Detection True Extrema Detected Extrema Inaccurate Keypoint Localization • The Problem: x Sampling

19. Description Applications Introduction Detection Inaccurate Keypoint Localization • The Solution: • Taylor expansion: • Minimize to find accurate extrema: • If offset from sampling point is larger than 0.5 - Keypoint should be in a different sampling point. Brown & Lowe 2002

20. Description Applications Introduction Detection • Function value at the extremun - • If (pixel values in range [0,1]) - keypoint is discarded. Low Contrast Keypoints => down to 729 Keypoints after min. contrast threshold.

21. Description Applications Introduction Detection Eliminating Edge Responses • The Problem: • “Edge“ keypoints are poorly determined. Point detection Point detection Point can move along edge => unstable.

22. Description Applications Introduction Detection Eliminating Edge Responses • The Solution: • Check Keypoints “cornerness”. Point constrained • High “cornerness”  No dominant principal curvature component.

23. Description Applications Introduction Detection • Principal curvature are proportional to eigenvalues of Hessian matrix: Finding “Cornerness” • Harris (1988) showed: • Threshold: if r < 10 - ratio is too great, keypoint discarded.

24. Description Applications Introduction Detection Stable Keypoints => down to 536 Keypoints after “cornerness” threshold.

25. Today’s Talk Orientation Description Applications Introduction Detection Detection Description Application Extracting Keypoints Distinctive Description Matching Keypoints • Extrema detection • Correct localization • Choosing robust keypoints only • Local invariant orientation • Building keypoint descriptor • Nearest-neighbor algorithm • Finding 3 matches • Least-square affine approximation  1st part of the talk 2nd part of the talk Last part of the talk DEMO

26. SIFT Algorithm Description Applications Introduction Detection • Scale-space extrema detection. • Keypoint localization. • Orientation assignment. • Keypoint descriptor. Representation invariant to Rotation.

27. Description Applications Introduction Detection Gradients • For each sample point we compute gradient’s magnitude and orientation:

28. Description Applications Introduction Detection • Create gradient histogram (36 bins) weighted by magnitude and Gaussian window ( is 1.5 times that of the scale of a keypoint) Keypoints Orientation • Any histogram peak within 80% of highest peak is assigned to keypoint (multiple assignments possible).

29. SIFT Algorithm Description Applications Introduction Detection • Scale-space extrema detection. • Keypoint localization. • Orientation assignment. • Keypoint descriptor. Distinctive (yet invariant) Keypoint Representation.

30. What do we have (and what don’t) Description Applications Introduction Detection • Do: • For each Keypoint we have assigned location, scale and orientation. • Provides invariance to these parameters. • Don’t: • Sufficient distinctiveness. • Invariance to other parameters, such as 3D viewpoint and change of illumination.

31. Description Applications Introduction Detection • Create 16 gradient histograms (8 bins) weighted by magnitude and Gaussian window ( is 0.5 times of the window) Keypoint Descriptor Keypoint Descriptor - 128 (4x4x8) element vector

32. Description Applications Introduction Detection Change of Illumination • Change of brightness => doesn’t effect gradients (difference of pixels value). • Change of contrast => doesn’t effect gradients (up to normalization). • Saturation (non-linear change of illumination) => affects magnitudes much more than orientation. => Threshold gradient magnitudes to 0.2 and renormalize.

33. Today’s Talk Orientation Description Applications Introduction Detection Detection Description Application Extracting Keypoints Distinctive Description Matching Keypoints • Extrema detection • Correct localization • Choosing robust keypoints only • Local invariant orientation • Building keypoint descriptor • Nearest-neighbor algorithm • Finding 3 matches • Least-square affine approximation   1st part of the talk 2nd part of the talk Last part of the talk

34. Object Recognition Description Applications Introduction Detection • For training images: • Extracting keypoints by SIFT. • Creating descriptors database. • For query images: • Extracting keypoints by SIFT. • For each descriptor - finding nearest neighbor in DB. • Finding cluster of at-least 3 keypoints. • Performing detailed geometric fit check for each cluster.

35. Description Applications Introduction Detection Keypoint Matching • Best candidate mach: Nearest neighbor. Problem: There are keypoints that do not have correct matches (background clutter/were not detected in training images)

36. Keypoints Matching Description Applications Introduction Detection closest neighbor found • Solution: • Threshold: < 0.8 closest neighbor from different object • Eliminates 90% of false matches while discarding less than 5% of correct matches.

37. Finding Nearest Neighbor Description Applications Introduction Detection • No good algorithm for high dimensional spaces. => Use Best-Bin-First (BBF) algorithm (Beis and Lowe, 1997). => Returns closest neighbor with high probability. Low cost - cutting off search after checking specific number of candidates. Good enough - we only need to show 0.8 ratio between first and second neighbor.

38. Clustering Description Applications Introduction Detection • In order to identify an object with high probability - we need more than one match: • We cluster 3 keypoints using Hough Transform.

39. Geometric Verification Description Applications Introduction Detection • Hough Transform found clusters of keypoints. • We would like to verify that these points do match geometrically to trained image. • In order to do that we use least-squares approach to find affine transformation from training image to query image. Now we can be pretty sure we got the right object!

40. Examples Description Applications Introduction Detection • Training images: • Query image: • Recognition (clutter, occlusion, illumination, etc.):

41. Examples Description Applications Introduction Detection • Training images: • Query image: • Recognition (different viewpoint, non-distinctive): Total time to recognize all object in both examples is less than 0.3 sec. (on 2GHz Pentium 4 processor)

42. Image Registration Description Applications Introduction Detection [Brown & Lowe 2003]

43. Examples Description Applications Introduction Detection • Real time Object-Recognition: Recognition DEMO • Motion Tracking: Nose DEMO

44. Summary Description Applications Introduction Detection Detection Description Application Extracting Keypoints Distinctive Description Matching Keypoints • Extrema detection • Correct localization • Choosing robust keypoints only • Local invariant orientation • Building keypoint descriptor • Nearest-neighbor algorithm • Finding 3 matches • Least-square affine approximation    1st part of the talk 2nd part of the talk Last part of the talk

45. Thank you