Features, Feature descriptors, Matching

1. Features, Feature descriptors, Matching Jana Kosecka George Mason University

2. Computer Vision Visual Sensing Images I(x,y) – brightness patterns • image appearance depends on structure of the scene • material and reflectance properties of the objects • position and strength of light sources MSRI Workshop, January 2005

3. What gives rise to images • photometric properties of the environment • geometric properties of the environment MSRI Workshop, January 2005

4. Basic ingredients Radiance – amount of energy emitted along certain direction Iradiance – amount of energy received along certain direction BRDF – bidirectional reflectance distribution Lambertian surfaces – the appearance depends only on radiance, not on the viewing direction Image intensity for a Lambertian surface MSRI Workshop, January 2005

5. Challenges MSRI Workshop, January 2005

6. Image Primitives and Matching Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point MSRI Workshop, January 2005

7. Image Primitives and Correspondence Difficulties – ambiguities, large changes of appearance, due to change of viewpoint, non-uniquess MSRI Workshop, January 2005

8. Matching - Correspondence radiance Lambertian assumption Rigid body motion Correspondence MSRI Workshop, January 2005

9. Local Deformation Models • Translational model • Affine model • Transformation of the intensity values taking into account occlusions and noise MSRI Workshop, January 2005

10. Matching and Correspondence Motivated by problems • Reconstruction of 3D scene from multiple views • Object recognition using (constellation of) features models Varieties • Small base-line matching • Wide base-line matching – large view point changes • For now assuming Lambertian assumption – appearance of a local surface patch is independent of the viewpoint MSRI Workshop, January 2005

11. Feature Tracking and Optical Flow • Translational model • Small baseline • RHS approximation by the first two terms of Taylor series • Brightness constancy constraint MSRI Workshop, January 2005

12. Feature Tracking and Optical flow • Integrate around over image patch • Solve MSRI Workshop, January 2005

13. Optical Flow, Feature Tracking Conceptually: rank(G) = 0 blank wall problem rank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates In reality: choice of threshold is involved MSRI Workshop, January 2005

14. Affine feature tracking Intensity offset Contrast change MSRI Workshop, January 2005

15. Optical Flow • Previous method - assumption locally constant flow • Alternative regularization techniques (locally smooth flow fields, • integration along contours) • Qualitative properties of the motion fields MSRI Workshop, January 2005

16. Point Feature Extraction • Compute eigenvalues of G • If smalest eigenvalue  of G is bigger than  - mark pixel as candidate • feature point • Alternatively feature quality function (Harris Corner Detector) MSRI Workshop, January 2005

17. Harris Corner Detector - Example MSRI Workshop, January 2005

18. Feature Selection • Compute Image Gradient • Compute Feature Quality measure for each pixel • Search for local maxima MSRI Workshop, January 2005 Feature Quality Function Local maxima of feature quality function

19. Feature Tracking • Translational motion model • Closed form solution • Build an image pyramid • Start from coarsest level • Estimate the displacement at the coarsest level • Iterate until finest level MSRI Workshop, January 2005

20. Coarse to fine feature tracking 0 1 2 • compute • warp the window in the second image by • update the displacement • go to finer level • At the finest level repeat for several iterations MSRI Workshop, January 2005

21. Tracked Features MSRI Workshop, January 2005

22. Wide baseline matching Point features detected by Harris Corner detector MSRI Workshop, January 2005

23. Region based Similarity Metric • Sum of squared differences • Normalize cross-correlation • Sum of absolute differences MSRI Workshop, January 2005

24. NCC score for two widely separated views NCC score MSRI Workshop, January 2005

25. ( ) Advanced matching techniques NCC - is not invariant with respect to image transformation 1. Selected salient image locations - points, pieces of countours 2. Associate Local photometric descriptors 3. Invariance to image transformations + illumination changes MSRI Workshop, January 2005

26. Summary of the approach • Very good results in the presence of occlusion and clutter • local information • discriminant greyvalue information • robust estimation of the global relation between images • for limited view point changes • Solution for more general view point changes • wide baseline matching (different viewpoint, scale and rotation) • local invariant descriptors based on greyvalue information MSRI Workshop, January 2005

27. Local descriptors Greyvalue derivatives Invariance to image rotation : differential invariants [Koenderink87] MSRI Workshop, January 2005

28. Feature Detection and Matching • Detection of interest points/regions • Harris detector (extension to scale and affine invariance) • Computation of descriptors for each point (e.g. diff. invariants, steerable filters, SIFT descriptor) • Similarity of descriptors (Euclidean distance, Mahalanobis Distance) MSRI Workshop, January 2005

29. Keypoint Detector and SIFT Descriptor • Each image is characterized by a set of scale-invariant keypoints and their associated descriptors [D. Lowe,2000] • Keypoints - extrema in DOG pyramid • Descriptor – 8 bin orientation histograms computed over 4 x 4 grid overlayed over pixel neighbourhood and stacked together to form a 128 dim feature vector MSRI Workshop, January 2005

30. SIFT Keypoints MSRI Workshop, January 2005

31. Overview • Scale invariance is not sufficient for large baseline changes • State of the art on affine invariant points/regions • Affine invariant interest points • Application to recognition MSRI Workshop, January 2005

32. Scale invariant interest points Invariant points + associated regions [Mikolajczyk & Schmid’01] multi-scale Harris points selection of points at the characteristic scale with Laplacian Courtesy of Schimd’01 MSRI Workshop, January 2005

33. Viewpoint changes • Locally approximated by an affine transformation detected scale invariant region projected region Courtesy of Schimd’01 MSRI Workshop, January 2005

34. Affine invariant Harris points • Localization & scale influence affine neighhorbood => affine invariant Harris points (Mikolajczyk & Schmid’02) • Iterative estimation of these parameters • localization– local maximum of the Harris measure • scale – automatic scale selection with the Laplacian • affine neighborhood – normalization with second moment matrix Repeat estimation until convergence • Initialization with multi-scale interest points MSRI Workshop, January 2005

35. Alternative features/descriptors • Affine invariant regions (Tuytelaars et al.’00) • ellipses fitted to intensity maxima • parallelogram formed by interest points and edges • Maximally stable regions (Matas et al. BMVC’02) • regions stable across large range of thresholds, connected • components of thresholded image • descriptors – rotationaly and affine invariant and color moments MSRI Workshop, January 2005

36. Feature Matches 33 correct matches Courtesy of Schimd’01 MSRI Workshop, January 2005

37. Pieces of Countour/Line descriprors • Select salient pieces using scale invariant detection techniques • Characterize either the intensity profile along contour/or local neighbourhood with sideness information – form the descriptor • Type of suitable salient regions depends of the class of objects • Computational model of visual attention can guide the process of selecting salient regions MSRI Workshop, January 2005

38. Additional changes of the appearance MSRI Workshop, January 2005