Fast multiresolution image querying

1. Fast multiresolution image querying CS474/674 – Prof. Bebis

2. Paper • Jacobs, A. Finkelstein, and D. Salesin, “Fast multiresolution image quering”, Proceedings of SIGGRAPH, pp. 277-286, 1995

3. Problem • Search an image database to retrieve images that are similar to a query image. “query by content” or “query by example” Typically, the K best matches are reported.

4. Challenges • What features to use? • How to tolerate image distortions? • How to organize the data? • How to search fast? • How to reduce storage requirements?

5. Image Distortions • This study considers two types of image distortion: • A low-resolution image from a scanner or video camera. • A rough sketch of the image painted by the user. painted low resolution target

6. Tolerating Image Distortions • Need to design an effective “image query metric” that can accommodate image distortions as well as distinguish the target image from the rest of the database. • The metric should be “tunable” to better account for the types of image distortions anticipated in the query image.

7. Tolerating Image Distortions (cont’d) • Traditional metrics based on the L1and L2norms cannot handle inexact matching and are time consuming. L1 Q: query T: target L2 • Experiments (i.e., this paper) using these metrics have • shown that the target image is in the highest 1% of the retrieved • images only 3% of the time.

8. Fast Retrieval • Retrieval should be fast enough to handle tens of thousands of images at interactive rates. Fast metric computation Efficient image representation Efficient database organization

9. Proposed Method: Key Ideas • Multi-resolution image decomposition using Haar wavelets. • Compute a “signature” for each image, based on (truncated and quantized) Haar wavelet coefficients. • “Signature” has low storage requirements.

10. Proposed Method: Key Ideas (cont’d) • Compute image similarity using a metric that compares how many significant wavelet coefficients the query has in common with potential targets. • Metric can be tuned (i.e., using statistical analysis) to accommodate specific image distortions. • Organize data properly to facilitate fast computation of the metric and speed-up search.

11. User Interface Returns 20 highest-ranked targets at interactive rates!. Can process a 128 x 128 image query on a database of 20,000 images in under 0.5 seconds*. *Faster processing times should be possible using current technology!

12. Why using wavelets? • The use of wavelets allows the resolutions of the query and target images to be different . • Wavelet decompositions are fast to compute and yield a small number of coefficients. • The signature can be extracted from a wavelet-compressed version of the image directly.

13. Components of the metric • Color space: • Experimented with RGB, HSV, and YIQ color spaces. • Wavelet transform was applied on each color channel separately. • YIQ gave the best performance (i.e., for their data).

14. Components of the metric • Wavelet type: • Haar wavelets are the fastest to compute and simplest to implement. • Other types of wavelets might give better results but at a higher cost.

15. Components of the metric (cont’d) • Decomposition type: • Experimented both with standard and non-standard decompositions for all three color spaces. • Standard decomposition worked best (i.e., both for scanned and painted queries).

16. Components of the metric (cont’d) • Truncation: • 128 x 128 image  1282 = 16,384 wavelet coefficients for each color channel! • Keep only the coefficients with largest magnitude. • Accelerates the search for a query. • Reduces storage requirements. • Improves discriminatory power of metric! • The 60 largest coefficients in each channel worked best for painted queries. • The 40 largest coefficients in each channel worked best for scanned queries.

17. Components of the metric (cont’d) truncated coefficients wavelet decomposition

18. Components of the metric (cont’d) • Quantization: • Quantize each of the retained coefficients into three levels: +1, 0 and -1 • Large positive coefficients are quantized to +1 • Large negative coefficients are quantized to -1 • Improves discriminatory power of metric! • The mere presence or absence of these coefficients appears to be more important than their precise magnitudes. • Improves speed and reduces storage requirements.

19. Components of the metric (cont’d) truncated coefficients truncated and quantized coefficients

20. Components of the metric (cont’d) • Normalization: • Basis functions are normalized so they become orthonormal to each other (see lecture slides on wavelets).

21. Wavelet-based metric • Suppose Q and T represent a single channel of the wavelet decomposition of the query and target images. • Let Q[0, 0] and T[0, 0] be the scaling function coefficients (i.e., average intensity of that channel). • Let and represent the truncated, quantized wavelet coefficients of Q and T (i.e., -1,0,1). (assume ) wi,j : weights (to be determined)

22. Simplifying the metric (cont’d) • Replace with (new metric was found to be as effective as the previous one)

23. Simplifying the metric (cont’d) • Group terms together into "buckets" so that only a small number of weights wi, jneeds to be determined experimentally. i,j

24. Simplifying the metric (cont’d) • Consider only the terms for which • Even faster computation. • Allows for a query without much detail to match a very detailed target image. i,j

25. Fast metric implementation(depends to data organization) • The majority of database images will not match the query. • It would be quicker to count the number of matching coefficients than the number of mismatching coefficients.

26. Fast metric implementation (cont’d)

27. Fast metric implementation (cont’d) • The term does not depend on the target image. • Ignore it for the purpose of ranking the different target images:

28. Example

29. Algorithm • Preprocessing • Perform a standard 2D Haar wavelet decomposition of every image in the database. (2) Store T[0,0] for each color channel and the indices and signs of the m wavelet coefficients of largest magnitude. (3) Organize the indices for all the images into a single data structure to optimize searching.

30. Algorithm (cont’d) • Querying • Perform the same wavelet decomposition on the query image. (2) Throw away all but the average color and the largest m coefficients. (3) Compute the score of each target image using the above equation.

31. Data Organization – Search Arrays • To optimize the search process, the m coefficients from every image are organized into a set of six 2D arrays (i.e. search arrays). • There is an array for every combination of sign (+ or -) and color channel (Y, I, and Q): contains a list of all images T having a large positive wavelet coefficient T[i, j] in color channel c.

32. Querying Using Search Arrays • Compute a score for each target image by looping through each color channel c. • Return top 20 matches

33. Querying Using Search Arrays (cont’d) • Steps • Compute the difference between the query’s average intensity in that channel Qc[0, 0] and those in the database. (2) For each of the m nonzero, truncated wavelet coefficients Qc[i, j], go through the list corresponding to Dc+[i, j] or Dc- [i, j] (i.e., depending on the sign of Qc[i, j]). (3) Update the score of each image found in those lists.

34. Weights wij • The function bin(i, j) groups different coefficients into a small number of bins (i.e., 6 bins per color channel): bin(i, j) = min(max(i, j), 5) • Each bin is weighted by some constant w[b] • Weights were determined using a statistical test (see paper).

35. Examples • Query examples using painted/scanned queries (ranks for database sizes: 1093 | 20,558)

36. Examples (cont’d) Interactive query examples using painted queries: (ranks for database sizes: 1093 | 20,558)

37. Some Results • Success rate: Lq : proposed metric Percentage of queries whose correct target was ranked among the top 1% of images in a database of 1093 images.

38. Some Results (cont’d) • Time requirements: Lq : proposed metric Average times to match a single query in a database of 1093/20,558 images.

39. Extension • V. Nikulin and G. Bebis, "Multiresolution Image Retrieval Through Fusion", SPIE Electronic Imaging (Storage and Retrieval Methods and Applications for Multimedia), San Jose, January 2004.