Locating and Describing Interest Points

1. 03/02/10 Locating and Describing Interest Points Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem Acknowledgment: Many keypoint slides from Grauman&Leibe 2008 AAAI Tutorial

2. What is “object recognition”?

3. 1. Identify a Specific Instance • General objects • Challenges: rotation, scale, occlusion, localization • Approaches • Geometric configurations of keypoints (Lowe 2004) • Works well for planar, textured objects

4. 1. Identify a Specific Instance • Faces • Typical scenario: few examples per face, identify or verify test example • What’s hard: changes in expression, lighting, age, occlusion, viewpoint • Basic approaches (all nearest neighbor) • Project into a new subspace (or kernel space) (e.g., “Eigenfaces”=PCA) • Measure face features • Make 3d face model, compare shape+appearance (e.g., AAM)

5. 2. Detect Instance of a Category • Much harder than specific instance recognition • Challenges • Everything in instance recognition • Intraclass variation • Representation becomes crucial

6. 2. Detect Instance of a Category • Template or sliding window • Works well when • Object fits well into rectangular window • Interior features are discriminative Schneiderman Kanade 2000

7. 2. Detect Instance of a Category • Parts-based Fischler and Elschlager 1973 Felzenszwalb et al. 2008

8. 3. Assign a label to a pixel or region • Stuff • Materials, object regions, textures, etc. • Approaches • Label patches + CRF • Segmentation + Label Regions

9. General Process of Object Recognition Specify Object Model Generate Hypotheses Score Hypotheses Resolution

10. General Process of Object Recognition Example: Template Matching Specify Object Model Intensity Template, at x-y Scanning window Generate Hypotheses Score Hypotheses Normalized X-Corr Resolution Threshold + Non-max suppression

11. General Process of Object Recognition Example: Keypoint-based Instance Recognition Specify Object Model A1 A3 B3 A2 Affine-variant point locations Generate Hypotheses Affine Parameters B1 B2 Score Hypotheses # Inliers Resolution Choose hypothesis with max score above threshold

12. General Process of Object Recognition Example: Keypoint-based Instance Recognition Specify Object Model A1 A3 B3 A2 Generate Hypotheses B1 B2 Today’s Class Score Hypotheses Resolution

13. Overview of Keypoint Matching 1. Find a set of distinctive key- points 2. Define a region around each keypoint A1 B3 3. Extract and normalize the region content A2 A3 B2 B1 4. Compute a local descriptor from the normalized region 5. Match local descriptors K. Grauman, B. Leibe

14. Main challenges • Change in position and scale • Change in viewpoint • Occlusion • Articulation

15. Goals for Keypoints Detect points that are repeatable and distinctive

16. Key trade-offs B3 A1 A2 A3 B2 B1 Localization More Repeatable More Points Robust to occlusion Works with less texture Robust detection Precise localization Description More Selective More Robust Minimize wrong matches Deal with expected variations Maximize correct matches

17. Keypoint Localization • Goals: • Repeatable detection • Precise localization • Interesting content K. Grauman, B. Leibe

18. Choosing interest points • If you wanted to meet a friend would you say • “Let’s meet on campus.” • “Let’s meet on Green street.” • “Let’s meet at Green and Wright.” • Corner detection • Or if you were in a secluded area: • “Let’s meet in the Plains of Akbar.” • “Let’s meet on the side of Mt. Doom.” • “Let’s meet on top of Mt. Doom.” • Blob (valley/peak) detection

19. Choosing interest points • Corners • “Let’s meet at Green and Wright.” • Peaks/Valleys • “Let’s meet on top of Mt. Doom.”

20. Many Existing Detectors Available Hessian & Harris [Beaudet ‘78], [Harris ‘88] Laplacian, DoG[Lindeberg ‘98], [Lowe 1999] Harris-/Hessian-Laplace[Mikolajczyk & Schmid ‘01] Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04] EBR and IBR [Tuytelaars & Van Gool ‘04] MSER[Matas ‘02] Salient Regions [Kadir & Brady ‘01] Others… K. Grauman, B. Leibe

21. Hessian Detector [Beaudet78] • Hessian determinant Ixx Iyy Ixy Intuition: Search for strongderivatives in two orthogonal directions K. Grauman, B. Leibe

22. Hessian Detector [Beaudet78] • Hessian determinant Ixx Iyy Ixy In Matlab: K. Grauman, B. Leibe

23. Hessian Detector – Responses [Beaudet78] Effect: Responses mainly on corners and strongly textured areas.

24. Hessian Detector – Responses [Beaudet78]

25. Harris Detector [Harris88] • Second moment matrix(autocorrelation matrix) Intuition: Search for local neighborhoods where the image content has two main directions (eigenvectors). K. Grauman, B. Leibe

26. Ix Iy Harris Detector [Harris88] • Second moment matrix(autocorrelation matrix) 1. Image derivatives Iy2 IxIy Ix2 2. Square of derivatives g(IxIy) g(Ix2) g(Iy2) 3. Gaussian filter g(sI) 4. Cornerness function – both eigenvalues are strong g(IxIy) 5. Non-maxima suppression har

27. Harris Detector – Responses [Harris88] Effect: A very precise corner detector.

28. Harris Detector – Responses [Harris88]

29. So far: can localize in x-y, but not scale

30. Automatic Scale Selection How to find corresponding patch sizes? K. Grauman, B. Leibe

31. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

32. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

33. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

34. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

35. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

36. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

37. What Is A Useful Signature Function? • Laplacian-of-Gaussian = “blob” detector K. Grauman, B. Leibe

38. Laplacian-of-Gaussian (LoG) • Local maxima in scale space of Laplacian-of-Gaussian s5 s4 s3 s2  List of(x, y, s) s K. Grauman, B. Leibe

39. Results: Laplacian-of-Gaussian K. Grauman, B. Leibe

40. Difference-of-Gaussian (DoG) • Difference of Gaussians as approximation of the Laplacian-of-Gaussian = - K. Grauman, B. Leibe

41. DoG – Efficient Computation • Computation in Gaussian scale pyramid Sampling withstep s4=2 s s s s Original image K. Grauman, B. Leibe

42. Results: Lowe’s DoG K. Grauman, B. Leibe

43. Orientation Normalization p 2 0 • Compute orientation histogram • Select dominant orientation • Normalize: rotate to fixed orientation [Lowe, SIFT, 1999] T. Tuytelaars, B. Leibe

44. Harris-Laplace [Mikolajczyk ‘01] • Initialization: Multiscale Harris corner detection s4 s3 s2 s Computing Harris function Detecting local maxima

45. Harris-Laplace [Mikolajczyk ‘01] • Initialization: Multiscale Harris corner detection • Scale selection based on Laplacian(same procedure with Hessian  Hessian-Laplace) Harris points Harris-Laplace points K. Grauman, B. Leibe

46. Maximally Stable Extremal Regions [Matas ‘02] • Based on Watershed segmentation algorithm • Select regions that stay stable over a large parameter range K. Grauman, B. Leibe

47. Example Results: MSER K. Grauman, B. Leibe

48. Available at a web site near you… • For most local feature detectors, executables are available online: • http://robots.ox.ac.uk/~vgg/research/affine • http://www.cs.ubc.ca/~lowe/keypoints/ • http://www.vision.ee.ethz.ch/~surf K. Grauman, B. Leibe

49. Local Descriptors • The ideal descriptor should be • Robust • Distinctive • Compact • Efficient • Most available descriptors focus on edge/gradient information • Capture texture information • Color rarely used K. Grauman, B. Leibe

50. Local Descriptors: SIFT Descriptor • Histogram of oriented gradients • Captures important texture information • Robust to small translations / affine deformations [Lowe, ICCV 1999] K. Grauman, B. Leibe