1 / 49

WISE: Large Scale Content-Based Web Image Search

WISE: Large Scale Content-Based Web Image Search. Michael Isard Joint with: Qifa Ke, Jian Sun, Zhong Wu Microsoft Research Silicon Valley. Query by Images. “ A picture is worth a thousand words . ”. What leaf ?. Artist ? Higher resolution?. © : Who else using this?. Bank web-site?.

yuri
Download Presentation

WISE: Large Scale Content-Based Web Image Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WISE: Large Scale Content-Based Web Image Search Michael IsardJoint with: Qifa Ke, Jian Sun, Zhong Wu Microsoft Research Silicon Valley

  2. Query by Images • “A picture is worth a thousand words.” What leaf ? Artist ? Higher resolution? ©: Who else using this? Bank web-site? Email ? Ad. in e-bay ? ……

  3. Partial-Duplicate Image Search • Given a query image, find its partial duplicates from a database of web-images

  4. Two Major Challenges • How to represent images • No text annotations or labels • Noise and modification • How to efficiently index and query images • Large number of images (millions) ? Index

  5. Image Representation: Bag-of-Words[CVPR’09, ICCV’09] 1: Feature extraction: Bundle Features 2: Quantization 3: Representation Detection [Lowe 2004, Matas et al 2002, Winder et al 2007] Code-book Bag-of-words[Sivic&Zisserman’2003] …… 3506 Normalization … 999,999 …… 1 206 Description [Lowe 2004, Winder et al 2007]

  6. …

  7. Matching query to database • Use an index • Each visual word has a ‘posting list’ • Lists every image containing the word • At query time • Look up the posting list for each query word • Merge lists to find candidate images • Partial match: don’t need every word to be present

  8. How much work to query? • Disk-based index, bottleneck is random reads • One seek per posting list • Also one seek per matching image • To fetch thumbnail etc • Keep as little information as possible in posting lists, to keep index size small

  9. Index Pipeline • Implemented in a large computer cluster • 256-nodes, using Dryad/DryadLINQ Content Chunk Feature Extractor Feature Quantizer Visual Words Image Crawler Bundled Features Indexer Media DB (Thumbnail, URL) Inverted Index Visual Word Index Crawler Local Features

  10. Query Pipeline Inverted Index GUI Results Media DB (Thumbnail, URL) Index Server Feature Extractor Feature Quantizer Visual Words Query Image Bundled Features Search Results (chunkID, imgID) …

  11. Bag-of-Words: Limitations • Quantization • Lost discriminative power • Sensitive to image variations and noises • Soft quantization[Philbin et al, CVPR 2008] • Hamming embedding [Jegou et al, ECCV 2008] Quantization …… ……

  12. Geometric verification • In practice, bag of words is too weak • Does not exploit any geometry • Post-process to check spatial layout of matching features • Requires a disk seek per image • Only used as a re-ranking step to shortlist of matched images

  13. 0.75 0.7 0.65 0.6 0.55 mAP 0.5 0.45 0.4 baseline (bag-of-words) 0.35 baseline + reranking 0.3 0.25 50000 200000 500000 1000000 Number of images Geometric Re-ranking Re-rank top 300 images

  14. Geometry in the index • Previous works: • Jegou et al ECCV 2008 • Try to match similar orientations and scales • Perdoch et al CVPR 2009 • Match oriented features more effectively • Still feature-by-feature • Global geometric consitency applied at the end

  15. Single Feature is Weak + +

  16. Neighboring Features ?

  17. Define Neighboring Features • Previous works • kNN voting [Sivic&Zisserman 2003] • Higher-order spatial features [Liu et al][Yuan et al][Tirilly et al][Quack et al] • Post geometric spatial verification [Lowe’2004][Chum et al 2007][Nister 2006][Philbin et al 2007]…… • Geometric Min-Hash [Chum et al 2009] • Challenges • Repeatable • Partial matching • Scalable: simple enough to build into index

  18. Define Neighboring Features DoG Features[Lowe 2004] MSER Features [Matas et al 2002] • point features • - repeatable • region features • - repeatable

  19. Define Neighboring Features DoG Features[Lowe 2004] MSER Features [Matas et al 2002] • point features • - repeatable • region features • - repeatable region groups points?

  20. Bundled Feature: Definition • Bundled Feature = A set of DOG features bundled by a MSER region MSER region DoG interest points

  21. Bundled Feature: Definition Bundled Features

  22. Matching Bundles: Membership Query bundle q = {qj} = { } Matched bundle p = { pi } Membership score: Voting weight:

  23. Matching Bundles: Membership Query bundle q = {qj} = { } Matched bundles p1, p2, p3 Membership score: p1 p3 p2

  24. Matching Bundles: Geometric Constraint y query candidate query candidate 5 5 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 1 order in query img: 1 < 2 < 3 <4 order in query img: 1 < 2 < 3 <4 order in target img: 1 < 3 < 4 < 5 matching order: 5>2>1 < 3 order inconsistency: 0 + 0 + 0 = 0 inconsistency: 1 + 1 + 0 = 2 Penalize inconsistent relative orders:

  25. Matching Bundles: Formulation • Bundle matching score: • Image matching score: membership geometric constraint • - Repeatable • - Partial matching • - Scalable?

  26. Inverted Index (without Bundles) …… Visual word … … … 27 … Posting 27 …… Image ID Image ID = 27 …… ……

  27. Inverted Index with Bundles …… Visual word … … … 27,[1,2,1] … Posting 27,[3,1,1] …… Image ID Bundle Bits Image ID = 27 9 bits 5 bits 5 bits p1 1 Bundle ID X-Order Y-Order 3 p2 27,[2,2,2] p3 2 …… ……

  28. Retrieval 9,2, [3,2,5] … 1,1, [2,5,9] 10, 1, [2,2,1] 3,1,[3,4,5] … 10, 1, [1,1,2] … … 12,1,[1.1.2] Query Image Iq Inverted index with bundle bits …… …… …… Top candidate images p1 p2 p3

  29. Experimental Settings • Image database: • 1M web images from query-click log • Ground truth partial duplicates • 780 known partial duplicate images in 19 groups • Baseline bag-of-words • Visual word vocabulary size = 1 M • Soft quantization factor = 4 • 500 features per image

  30. Partial Duplicate Example ……

  31. Partial Duplicate Example ……

  32. Example Query Results Query Challenging cases

  33. Evaluation: Precision-Recall • A query returns N images • T : correct matches • A : expected matches Precision Recall

  34. Comparison: Precision-Recall Query image: Baseline bag-of-words (started from 13th) Bundled features (started from 13th)

  35. More Precision-Recall Comparisons

  36. Evaluation: mAP • Average Precision (AP) for one query: • Area under Precision-Recall curve • mAP: mean of AP’s from all testing queries AP

  37. mAP: Baseline Bag-of-Words

  38. mAP: Hamming Embedding (HE)

  39. mAP: Bundle (Membership)

  40. mAP: Bundle (both terms) 26% 40%

  41. mAP: Bundle + HE 49%

  42. Bundle VS. Geometric Re-ranking Re-rank top 300 images

  43. Bundle + Geometric Re-ranking Re-rank top 300 images 24% 77%

  44. More Results

  45. Failure Case

  46. Demo Setup Client Web Server Document Server Index Servers 6 million images

  47. Demo Query image Results

  48. Conclusion • Bundle feature • More discriminative • Enforce spatial constraints while traversing index • Partial match • Scalable: built into index 9 bits 5 bits 5 bits Bundle ID X-Order Y-Order

  49. Thanks!

More Related