1 / 30

Image Retrieval with Geometry-Preserving Visual Phrases

Image Retrieval with Geometry-Preserving Visual Phrases. Yimeng Zhang, Zhaoyin Jia and Tsuhan Chen Cornell University. Similar Image Retrieval. …. Image Database. Ranked relevant images. Bag-of-Visual-Word (BoW) . Images are represented as the histogram of words

mercia
Download Presentation

Image Retrieval with Geometry-Preserving Visual Phrases

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Retrieval with Geometry-Preserving Visual Phrases Yimeng Zhang, Zhaoyin Jia and Tsuhan Chen Cornell University

  2. Similar Image Retrieval … Image Database Ranked relevant images

  3. Bag-of-Visual-Word (BoW) Images are represented as the histogram of words Similarity of two images: cosine similarity of histograms Length: dictionary size …

  4. Geometry-preserving Visual Phrases length-k Phrase:: k words in a certain spatial layout … Bag of Phrases: (length-2 phrases) …

  5. Phrases vs. Words Relevant Irrelevant Word Word Length-2 Length-2 Length-3 Length-3

  6. Previous Works

  7. Geometry Verification Only on top ranked images Searching Step with BoW Encode Spatial Info … Post-processing(Geometry Verification)

  8. Modeling relationship between words Co-occurrences in Entire image [L. Torresani, et al, CVPR 2009] No spatial information Phrases in a local neighborhoods [J. Yuan et al, CVPR07][Z. Wu et al., CVPR10] [C.L.Zitnick, Tech.Report 07] No long range interactions, weak geometry Select a subset of phrases [J. Yuan et al, CVPR07] Discard a large portion of phrases (length-2 Phrase) Previous works: reduce the number of phrases Dimension: exponential to # of words in Phrase Our work: All phrases, Linear computation time … …

  9. Approach

  10. Overview Similarity Measure BoP BoW [Zhang and Chen, 09] 2. Large Scale Retrieval This Paper Min-hash Inverted Files Min-hash Inverted Files

  11. Co-occurring Phrases Only consider the translation difference A B D C F E F A A B C D A F E F [Zhang and Chen, 09]

  12. Co-occurring Phrase Algorithm A B A 3 2 1 0 -1 -2 -3 -4 D C # of co-occurring length -2 Phrases: F DF EF F E F B A 1 +1 =5 F A C A F A A B C D -2 -1 0 1 2 3 4 A F Offset space E F [Zhang and Chen, 09]

  13. Relation with the feature vector … … … … same as BOW!!! # of co-occurring length-k phrases Inner product of the feature vectors M: # of corresponding pairs, in practice, linear to the number of local features

  14. Inverted Index with BoW Avoid comparing with everyimage Inverted Index … … … … … +1 Score table

  15. Inverted Index with Word Location … … I1 … Assume same word only occurs once in the same image,  Same memory usage as BoW … … … …

  16. Score Table Compute # of Co-occurring Phrases: Compute the Offset Space BoW I1 I2 In … BoP

  17. Inverted Files with Phrases … I1 I10 Inverted Index … 0,-1 1,-1 … -1,-1 I8 -1,0 0,0 1,0 … 0,1 … I5 wi … … +1 +1 +1 +1 Offset Space … … … …

  18. Final Score In I2 I1 5 2 4 … Offset Space 8 3 10 2 2 2 1 1 Final similarity scores

  19. Overview BoP BoW Min-hash Inverted Files Min-hash Inverted Files Less storage and time complexity

  20. Min-hash with BoW Probability of min-hash collision (same word) = Image Similarity I I’

  21. Min-hash with Phrases Probability of k min-hash collision with consistent geometry (Details are in the paper) I 3 2 1 0 -1 -2 -3 -4 I’ -3 -2 -1 0 1 2 Offset space

  22. Other Invariances Add dimension to the offset space Increase the memory usage Image I Image I’ [Zhang and Chen, 10]

  23. Variant Matching Local histogram matching

  24. Evaluation BoW + Inverted Index vs. BoP + inverted Index BoW + Min-hash vs. BoP + Min-hash Post-processing methods: complimentary to our work

  25. Experiments –Inverted Index • 5K Oxford dataset (55 queries) • 1M flicker distracters Philbin, J. et al. 07

  26. Example Precision-recall curve Higher precision at lower recall BoW BoW BoW Precision BoP BoP Precision Recall Recall

  27. Comparison Mean average precision: mean of the AP on 55 queries BoP+RANSAC BoP BoW BoW+RANSAC • Outperform BoW (similar computation) • Outperform BoW+RANSAC(10 times slower on 150 top images) • Larger improvement on smaller vocabulary size

  28. +Flicker 1M Dataset • Computational Complexity RANSAC: 4s on top 300 images

  29. Experiment - min-hash University of Kentucky dataset Minhash with BoW: [O. Chum et al., BMVC08]

  30. Conclusion • Encode more spatial information into the BoW • Can be applied to all images in the database at the searching step • Same computational complexity as BoW • Better Retrieval Precision than BoW+RANSAC

More Related