Dirk Nuemann and Karl R. Gegenfurtner 2002

1. Perception Based Image Retrieval Dirk Nuemann and Karl R. Gegenfurtner 2002

2. Introduction Image indexing system based on known properties of the early stages of human vision. Find a set of images that is perceptually similar to some target image. Database of 60,000 photographs. Washington University in St. Louis Media & Machines Lab

3. Color and Spatial Indexing For each image the distribution of the features hue, luminance and spatial frequency were determined. • Summarized as three histograms (feature vectors) • Similarity of two images is the distance between the feature vectors. Modeling of the appropriate image features was done in DKL color space. • Chromaticity and luminance stored as two separate indices. • Information about Fourier energy distribution was stored in a third histogram. Washington University in St. Louis Media & Machines Lab

4. Color and Luminance Indexing Color distribution of an image plotted w.r.t. the coordinates of the color opponent axes in DKL color space. Washington University in St. Louis Media & Machines Lab

5. Color and Luminance Indexing Conversion from Cartesian color opponent coordinates to polar coordinates: (Color angle) (Distance from neutral gray) Washington University in St. Louis Media & Machines Lab

6. Color and Luminance Indexing Color histograms were created by dividing the chromaticity plane into 127 logarithmic-radial bins. Washington University in St. Louis Media & Machines Lab

7. Color and Luminance Indexing The size of the bins reflect the granularity of higher order perception: • For saturated colors, the resolution of hue is much finer than for unsaturated colors. 6 rings are used to discriminate saturation. The ring a color belongs to is calculated using the logarithm of the saturation. Color values exceeding 95% or falling below 5% of the maximally possible luminance value are classified as white and black respectively. 129 bins, 64 for the most saturated colors. Washington University in St. Louis Media & Machines Lab

8. Color and Luminance Indexing To calculate the color and luminance indices: • The color distribution of each image as mapped into these 129 bins. • For each image two vectors were stored. • The frequency of the color tones f. • The average luminance level of the pixels in each bin l. • If bin is empty l is set to zero. Washington University in St. Louis Media & Machines Lab

9. Color and Luminance Indexing Frequency of the color bins for image with hats. The brightness of the bins is proportional to their frequency in the image. The most frequent is shown in white. Washington University in St. Louis Media & Machines Lab

10. Color and Luminance Indexing Color tones belonging to each bin are plotted with the average luminance of that bin (hats image). Washington University in St. Louis Media & Machines Lab

11. Fourier Index 2D discrete Fourier transformation (DFT) was used to determine the distribution of Fourier energy across different orientations. Done only in the luminance dimension of the DKL color space. DFT determines the amplitude for a set of 2D sine waves. These basis functions can be characterized by a single complex parameter. The polar coordinates of this parameter give the orientation and spatial frequency of the 2D sine waves. The resulting Fourier spectrum was divided into radial logarithmic bins. Washington University in St. Louis Media & Machines Lab

12. Fourier Index 126 bins Each bin represents contrasts of distinct orientation and frequency ranges. Spatial frequency increases from the center to the edge of the chart. The brightness of a bin is proportional to the logarithm of the average energy within the bin. Highest energy is white. (for hats image). Washington University in St. Louis Media & Machines Lab

13. Distance metrics Euclidean distance is used to compare the luminance and Fourier indices. For chromaticity, the similarity between two images is given by the sum of the minimum of the corresponding bin frequencies. Interpreted as the proportion the two color distributions share. 0.75 means that for 75% of the pixels in one image, there exists pixels in the other image which fall into corresponding color bins. Washington University in St. Louis Media & Machines Lab

14. Distance metrics Washington University in St. Louis Media & Machines Lab

15. Demo Washington University in St. Louis Media & Machines Lab

16. Psychophysical Evaluation Measure the relationship between the perceived similarity and the computed similarity. • Done for each of the three indices and for combinations of the three indices. Two-alternative forced choice configuration. Similarity (judged) of an image to the query image can be defined as the probability of selecting that image. Washington University in St. Louis Media & Machines Lab

17. Psychophysical Evaluation 900 query images randomly selected from database. For each of these the 2000 best matching images for each relevant index of index combination were retrieved. The images with rank 2, 20, 200 and 2000 were selected for the experiment. Self paced without decision time limits. Washington University in St. Louis Media & Machines Lab

18. Experiment 1 For each of three distractor images, the rate of preferring the target is shown on the y axis. Changing distractor similarity does not change the gradient but shifts the curves by constant amounts. Washington University in St. Louis Media & Machines Lab

19. Experiment 2 Washington University in St. Louis Media & Machines Lab

20. Experiment 3 5% increase in judgments when spatial information included. Further 4% increase when luminance is included. Washington University in St. Louis Media & Machines Lab

21. Questions ? Washington University in St. Louis Media & Machines Lab