Global and Efficient Self-Similarity for Object Classification and Detection

1. Global and Efficient Self-Similarity for Object Classification and Detection Thomas Deselaers and Vittorio Ferrari CALVIN group Computer Vision Laboratory ETH Zurich Switzerland CVPR 2010

2. Conventional Image Descriptors Measure direct image properties gradients colors

3. Self-Similarity vs Conventional Descriptors [Shechtman, Irani CVPR 07] Assumption of conventional image descriptors There is a direct visual property shared by images of objects of the same class (e.g. colors, gradients, …). This property can be used to compare images. Self-similarity: Indirect property: geometric layout of repeating patches within an image More general property

4. Local Self-Similarity Descriptors [Shechtman, Irani CVPR 07]

5. Using Local Self-Similarity Descriptors Applications: object recognition, image retrieval, action recognition • Ensemble matching [Shechtman CVPR 07] • Nearest neighbor matching [Boiman CVPR 08] • Bag of local self-similarities [Gehler ICCV09, Vedaldi ICCV09, Hörster ACMM08, Lampert CVPR09, Chatfield ICCV09 WS] Compute LSS descriptors for an image Assign the LSS descriptors to a codebook Represent the image as a histogram of LSS descriptors

6. Self-Similarity goes Global Capture long-range self-similarities and their spatial arrangement

7. Self-Similarity goes Global Capture long-range self-similarities and their spatial arrangement

8. Global Self-Similarity Tensor compute self-similarity between all pairs of pixels 4D self-similarity tensor Note: local self-similarities included

9. Problems with the GSS Tensor 11 11 300 500 ∼ 20h ∼ 80GB Aim: Reduce both Computation time: Memory requirement:

10. Outline • Efficient global self-similarity tensor • Global self-similarity descriptors • Bag of correlation surfaces • Self-similarity hypercubes • Detection with self-similarity hypercubes • Efficient sliding window • Efficient subwindow search • Experiments • Global self-similarity better than local self-similarity • Complementary to conventional descriptors • Object detection possible

11. Efficient Global Self-Similarity Tensor Find an efficient approximation to If two patches are assigned to the same prototype, they are similar Quantize patches according to codebook Reduces runtime to speedup: 750

12. Efficient Global Self-Similarity Two patches are only similar if they are assigned to the same prototype Reduces memory to reduction:

13. Patch Prototype Codebooks Remember: Self-similarity encodes image content indirectly Image-specific codebooks can be smaller than conventional ones see paper for more generic codebooks and extensive evaluation

14. Global Self-Similarity Descriptors • So far: • Compact GSS computed efficiently • Now: • Descriptors that can be used in machine learning classifiers • Fixed dimensionality • Compact representation • Self-similarity hypercubes: now • Bag of correlation surfaces: only in the paper

15. Self-Similarity Hybercubes SSH of size

16. SSHs for Detection • Computing SSH naïvely requires operations • Sliding windows has to evaluate many windows operations

17. Efficient Computation of SSHs Compute integral self-similarity tensor: can be obtained using 16 lookups in  160000 operations to compute SSH for an image window  ∼5000x speedup

18. Efficient Subwindow Search for SSH • Derive an upper bound on the score of a set of windows • Section 5.2 in our paper • Similar to [Lampert PAMI09]

19. Experiments: Object classification PASCAL 07 objects • 9608 cropped images of objects from PASCAL 07 • 20 classes Task: Classify each test image into one of 20 classes Model: Linear SVM Train: train+val Test: test

20. Classification on the PASCAL 07 objects set classification accuracy [%] + GSS outperform LSS + Self-Similarity is truly complementary to conventional descriptors

21. Experiments: Object detection e.g. [Ferrari CVPR07, Maji CVPR09] ETHZ Shape Classes • 255 images • 5 classes (apple logos, bottles, giraffes, mugs, swans) Task: Detect objects in images Detector: Linear SVM, sliding windows

22. Detection Results DR at FPPI 0.4 } bottles giraffes SSH + SSH outperforms BOLSS + it is possible to use GSS for detection with good results apple logos swans mugs DR at 0.5 PASCAL overlap } BoLSS Comparison results (avg): [Ferrari CVPR07]: 71.9 [Maji CVPR09]: 93.2 … many more FPPI 0.4

23. Runtimes for Computing Descriptors • 200x200 image • GSS tensor • directly: 5512s (∼1.5 hours) • using our method: 81s (∼1.5 minutes) • Computing descriptors: few seconds • Our method: 70x speedup • For Reference: • GIST: 0.4s • BOLSS: 0.7s

24. Runtimes for Detection June 2014 Given the prototype assignment map (80s) (once only) SSH sliding window: 30s/img (once per class) For Comparison • Computing direct GSS tensor for 25000 windows: 4 years/img Speedup: ∼1 million ⇒ Using our methods, GSS can be used for object detection For Reference: • Felzenszwalb PAMI 09: 5s.

25. Feasible Global and Efficient Self-Similarity for Object Classification and Detection Thomas Deselaers and Vittorio Ferrari CALVIN group Computer Vision Laboratory ETH Zurich Switzerland CVPR 2010

26. Conclusion • self-similarity should be considered globally • Global self-similarity performs better than local self-similarity • truly complementary to conventional descriptors • global self-similarity is feasible • efficient computation of self-similarity • two descriptors based on self-similarity • global self-similarity for detection • code will be available soon

27. Thank you for your attention Thomas Deselaers and Vittorio Ferrari Global and Efficient Self-Similarity for Object Classification and Detection Code will be available http://www.vision.ee.ethz.ch/~calvin

28. Thank you for your attention Thomas Deselaers and Vittorio Ferrari Global and Efficient Self-Similarity for Object Classification and Detection Code will be available http://www.vision.ee.ethz.ch/~calvin