Bag of Features Approach: recent work, using geometric information

1. Bag of Features Approach: recent work, using geometric information

2. Problem • Search for object occurrences in very large image collection

3. 2 sub problems • Object Category Recognition and Specific Object Recognition

4. Motivation • Look for product information • Look for similar products

5. Related work on large scale image search • Most systems build upon the BoF framework [Sivic & Zisserman 03] • Large (hierarchical) vocabularies [NisterStewenius 06] • Improveddescriptorrepresentation [Jégou et al 08, Philbin et al 08] • Geometry used in index [Jégou et al 08, Perdoc’h et al 09] • Query expansion [Chum et al 07] • … • Efficiency improved by: • Min-hash and Geometrical min-hash [Chum et al. 07-09] • Compressing the BoF representation [Jégou et al. 09]

6. Local Features - SIFT

7. Creating a visual vocabulary 1 2 3 4

8. Inverted Index Index construction Searching

9. Use geometry • Possible directions: • Change/optimize spatial verification stage • Insert a new geometric information to the index • Ordered BOF • Bundled features • Visual phrases • Change the searching algorithm

10. Survey for today • Spatial Bag-of-features [Cao, CVPR2010] • Image Retrieval with Geometry-Preserving Visual Phrases [Zhang Jia Chen, CVPR2011] • Smooth Object Retrieval using a Bag of Boundaries [ArandjeloviZisserman, ICCV2011]

11. Spatial BOF • Basic idea:

12. Spatial BOF • Constructing linear and circular ordered bag-of-features:

13. Spatial BOF • Translation invariance:

14. Spatial BOF • Pros: • Gets better performance than BOF+RANSAC for large scale dataset* • Same format as standard BOF • Cons: • Is dataset dependent because of need of training • Do not present the results for large scale dataset with transfer learning from another dataset • Future work • Check it with cross training for large dataset. Otherwise, it is not worth working further.

15. Geometry-Preserving Visual Phrases • Basic idea:

16. Geometry-Preserving Visual Phrases • Representation • Quantize image to 10x10 grid • Histogram of GVPs of length k • GVP dictionary size is “choose k from N visual words”

17. Geometry-Preserving Visual Phrases • Pros: • Outperforms BOV + RANSAC • Cons: • Only translation invariant because of memory • Future work

18. BOF for smooth objects Idea: Gradient Segment The information used for retrieval Query object

19. BOF for smooth objects Results:

20. BOF for smooth objects Segmentation phase • Over segmentation with super-pixels • Classification of super-pixels: • 3208 feature vector (median(Mag(Grad)), 4 bits, color histogram, BOF) • SVM • Post-processing

21. BOF for smooth objects Boundary description phase: • Sample points on the boundary • Calculate HoG at each point in 3 scales 340 dimensional L2 normalized vector * The descriptor is not rotation invariant

22. BOF for smooth objects Retrieval procedure: • Boundary descripors are quantized (k=10k) • Standard BOF scheme* • Spatial verification for top 200 with loose affine homography (errors up to 100pixs) * No spatial information is recorded in the histogram

23. BOF for smooth objects • Pros: • Solves the smooth object retrieval problem • Fast • Cons: • Is dataset dependent because of need of training • Limited to objects with “solid” materials – segmentation has to catch the object’s boundary • Future work • Eliminate the training step

24. Summary • There is an active research in the field of CBIR to exploit geometry information. • Each method with its limitations • Still no widely accepted solution • Like spatial verification with RANSAC