Special Topic on Image Retrieval

1. Special Topic on Image Retrieval Local Feature Matching Verification

2. Geometric Verification • Motivation • Remove false matches by checking geometric consistency Red line: geometric consistent match Blue line: geometric inconsistent match

3. Global Verification: RANSAC • Take RANSAC as an example • Check geometric consistency from matched feature pairs. Random sampling

4. Local Geometric-Verification • Locally nearest neighbors （Video Goole, cvpr’03） • Matched regions should have a similar spatial layout. • For each match define its search area • Region in the search area that also matches casts a vote for the image • Reject matches with no support • Drawback • Sensitive to clutter

5. Hamming Embedding (ECCV’08) • Introduced as an extension of BOV [Jegou 08] • Combination of • A partitioning technique (k-means) • A binary code that refine the descriptor • Representation of a descriptor x • Vector-quantized to q(x) as in standard BOV • short binary vector b(x) for an additional localization in the Voronoi cell • Two descriptors x and y match iif

6. Hamming Embedding • Binary signature generation • Off-line learning • Random matrix generation • Descriptor projection and assignment • Median values of projected descriptors • On-line binarization • Quantization assignment • Descriptor projection • Computing the signature:

7. Local Geometric-Verification • Bundled feature (CVPR’09) • Group local features in local MSER region. • Increase discriminative power of visual words. • Allowed to have large overlap error. • Bundle comparison: • Mm(q; p): number of common visual words between two bundles • Mg(q; p): inconsistency of geometric order in x- and y- direction. • Drawbacks: Infeasible for rotated bundles.

8. Local Geometric-Verification Bundled feature (CVPR’09) • Visual words are bundled in MSER regions. • Spatial consistency for bundled features is utilized to weight visual words. # of shared visual words • Great performance for partial-dup detection in over 1 M database • Drawbacks: Infeasible for rotated bundles. Spatial consistency • Z. Wu, J. Sun, and Q. Ke, “Bundling Features for Large Scale Partial-Duplicate Web Image Search,” CVPR 09

9. Global Verification: RANSAC • RANSAC: remove outliers by inlier classification • Inliers: true matched features • Outliers: false matched features • Assumption of RANdomSAmpleConsensus (RANSAC) • The original data consists of inliers and outliers. • A subset of inliers can estimate a model to optimally explain the inliers. • Estimate the affine transformation by RANSAC • Procedure: Iteratively select a random subset as hypothetical inliers • A model is fitted to the hypothetical inliers. • All other data are tested against the fitted model for inlier classification. • The model is re-estimated from all hypothetical inliers. • The model is evaluated by estimating the error of the inliers relative to the model. • Drawbacks: Computationally expensive, not scalable Fischler, et al., RANdomSAmpleConsensus: a paradigm for model fitting with applications to image analysis and automated cartography, Comm. of the ACM, 24:381-395, 1981

10. Spatial Coding for Geometric Verification (ACM MM’ 10) • Motivation • Encode local features’ relative positions into compact binary maps • Check spatial consistency of local matches for geometric verification • Spatial coding maps • Relative spatial positions between local features. • Very efficient and high precision Zhou & Tian, Spatial Coding for large scale partial-duplicate image search. ACM Multimedia 2010.

11. Spatial Map Generation • In previous case, each quadrant has one part • Consider each quadrant is uniformly divided into two parts. = Rotate 45 degree counterclockwise

12. Spatial Map Generation • Generalized spatial map: GX and GY • Each quadrant is uniformly divided into r parts. X-map X-map X-map Y-map Y-map Y-map k=0 k=1 … … k=r-1

13. Generalized Spatial Coding • Spatial coding maps: • Each quadrant uniformly divided into r parts. • Decompose the division into r sub-division. • Rotate each sub-division to align the axis. New featurelocations after rotation ： Generalized spatial maps：

14. Spatial Verification • Verification with spatial maps GX and GY • Compare the spatial maps of matched features: • k=0, …, r-1; i, j=1, …, N; N: number of matched features • Find and delete the most inconsistent matched pair, recursively: Vx: inconsistent degree in X-map Vy: inconsistent degree in Y-map Identify i* and remove

15. 5 y 1 2 1 2 3 3 4 4 x 5

16. Geometric Verification with Coding Maps 5 1 y 2 1 2 3 3 4 4 x 5 SUM

17. 4 4 4 Image Plane Division (TOMCCAP’ 10) 3 3 3 2 2 2 1 1 1 5 5 5 4 4 4 (c) (a) (b) 3 3 3 2 2 2 5 5 5 1 1 1 (f) (d) (e)

18. 4 Geometric Square Coding 3 2 • Coordinate adjustment • Square coding map 1 5 4 3 2 5 Generalized map： 1

19. Geometric Fan Coding • Fan coding maps • Coordinate adjustment • Generalized coding maps

20. Geometric Verification • Compare the fan coding maps of matched features: • Inconsistency measurement from geometric fan coding: • Inconsistency measurement from geometric square coding: • Inconsistency matrix: