1 / 41

Spectral Hashing

Spectral Hashing. Y. Weiss (Hebrew U.) A. Torralba (MIT) Rob Fergus (NYU). What does the world look like?. Motivation. High level image statistics. Object Recognition for large-scale search . Semantic Hashing. [Salakhutdinov & Hinton, 2007]. Query Image. Semantic Hash Function.

niveditha
Download Presentation

Spectral Hashing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Spectral Hashing Y. Weiss (Hebrew U.) A. Torralba (MIT) Rob Fergus (NYU)

  2. What does the world look like? Motivation High level image statistics Object Recognition for large-scale search

  3. Semantic Hashing [Salakhutdinov & Hinton, 2007] Query Image Semantic HashFunction Address Space Binary code Images in database Query address Semantically similar images Quite differentto a (conventional)randomizing hash

  4. 1. Locality Sensitive Hashing • Gionis, A. & Indyk, P. & Motwani, R. (1999) • Take random projections of data • Quantize each projection with few bits 101 0 Gist descriptor 1 0 No learning involved 1 1 0

  5. Toy Example • 2D uniform distribution

  6. 2. Boosting • Modified form of BoostSSC [Shaknarovich, Viola & Darrell, 2003] • Positive examples are pairs of similar images • Negative examples are pairs of unrelated images 0 Learn threshold & dimension for each bit (weak classifier) 0 1 1 0 1

  7. Toy Example • 2D uniform distribution

  8. 3. Restricted Boltzmann Machine (RBM) • Type of Deep Belief Network • Hinton & Salakhutdinov, Science 2006 Hidden units Units are binary & stochastic SingleRBMlayer Symmetric weights W Visible units • Attempts to reconstruct input at visible layer from activation of hidden layer

  9. Multi-Layer RBM: non-linear dimensionality reduction Output binary code (N dimensional) N Layer 3 w3 256 256 Layer 2 w2 512 512 Layer 1 w1 512 Linear units at first layer Input Gist vector (512 dimensions)

  10. Toy Example • 2D uniform distribution

  11. 2-D Toy example: 3 bits 7 bits 15 bits Distance from query point Red – 0 bits Green – 1 bit Black – >2 bits Blue – 2 bits Query Point

  12. Toy Results Distance Red – 0 bits Green – 1 bit Blue – 2 bits

  13. Semantic Hashing [Salakhutdinov & Hinton, 2007] Query Image Semantic HashFunction Address Space Binary code Images in database Query address Semantically similar images Quite differentto a (conventional)randomizing hash

  14. Spectral Hash Query Image SpectralHash Non-lineardimensionality reduction Address Space Binary code Images in database Real-valuedvectors Query address Semantically similar images Quite differentto a (conventional)randomizing hash

  15. Spectral Hashing (NIPS ’08) • Assume points are embedded in Euclidean space • How to binarize so Hamming distance approximates Euclidean distance? Ham_Dist(10001010,11101110)=3

  16. Spectral Hashing theory • Want to min YT(D-W)Y subject to: • Each bit on 50% of time • Bits are independent • Sadly, this is NP-complete • Relax the problem, by letting Y be continuous. • Now becomes eigenvector problem

  17. Nystrom Approximation • Method for approximating eigenfunctions • Interpolate between existing data points • Requires evaluation of distance to existing data cost grows linearly with #points • Also overfits badly in practice

  18. What about a novel data point? • Need a function to map new points into the space • Take limit of Eigenvalues as n\inf • Need to carefully normalize graph Laplacian • Analytical form of Eigenfunctions exists for certain distributions (uniform, Gaussian) • Constant time compute/evaluate new point • For uniform: Only depends on extent of distribution (b-a)

  19. Eigenfunctions for uniform distribution

  20. The Algorithm Input: Data {xi} of dimensionality d; desired # bits, k • Fit a multidimensional rectangle to the data • Run PCA to align axes, then bound uniform distribution • For each dimension, calculate k smallest eigenfunctions. • This gives dkeigenfunctions. Pick ones with smallest k eigenvalues. • Threshold eigenfunctions at zero to give binary codes

  21. 1. Fit Multidimensional Rectangle • Run PCA to align axes • Bound uniform distribution

  22. 2. Calculuate Eigenfunctions

  23. 3. Pick k smallest Eigenfunctions Eigenvalues e.g. k=3

  24. 4. Threshold chosen Eigenfunctions

  25. Back to the 2-D Toy example 3 bits 7 bits 15 bits Distance Red – 0 bits Green – 1 bit Blue – 2 bits

  26. 2-D Toy Example Comparison

  27. 10-D Toy Example

  28. Experimentson Real Data

  29. Input Image representation: Gist vectors • Pixels not a convenient representation • Use Gist descriptor instead (Oliva & Torralba, 2001) • 512 dimensions/image (real-valued  16,384 bits) • L2 distance btw. Gist vectors not bad substitute for human perceptual distance NO COLOR INFORMATION Oliva & Torralba, IJCV 2001

  30. LabelMe images • 22,000 images (20,000 train | 2,000 test) • Ground truth segmentations for all • Assume L2 Gist distance is true distance

  31. LabelMe data

  32. Extensions

  33. How to handle non-uniform distributions

  34. Bit allocation between dimensions • Compare value of cuts in original space, i.e. before the pointwise nonlinearity.

  35. Summary • Spectral Hashing • Simple way of computing good binary codes • Forced to make big assumption about data distribution • Use point-wise non-linearities to map distribution to uniform • Need more experiments on real data

  36. Overview • Assume points are embedded in Euclidean space (e.g. output from RBM) • How to binarize the space so that Hamming distance between points approximates L2 distance?

  37. Semantic Hashing beyond 30 bits

  38. Strategies for Binarization • Deliberately add noise during backprop - forces extreme values to overcome noise 0 1 0 1

More Related