160 likes | 250 Views
Using Artificial Queries to Evaluate Image Retrieval. Nicholas R. Howe Department of Computer Science Cornell University. How Do We Compare Image Retrieval Algorithms?. Different research groups use images from different sources. Image sets are of different sizes. Tasks are different.
E N D
Using Artificial Queries to Evaluate Image Retrieval Nicholas R. Howe Department of Computer Science Cornell University
How Do We CompareImage Retrieval Algorithms? • Different research groups use images from different sources. • Image sets are of different sizes. • Tasks are different. • Each researcher identifies set of queries and targets through subjective criteria. • Can’t share keys because image sets are not standard. Answer: Badly! Workshop on Content-Based Access of Image and Video Libraries
How It’s Usually Done • Each researcher tests a proposed algorithm against a few baselines. • e.g., Color Histograms. • No data to compare latest techniques. • Test sets are different. • Implementation of baselines may differ also. Workshop on Content-Based Access of Image and Video Libraries
Some Difficulties • Given a query, which target is most relevant? • Context will determine answer. ? Workshop on Content-Based Access of Image and Video Libraries
What Should a Good Test Do? • Provide comparable results even with different image sets. • Offer insight into the behavior of different retrieval algorithms. • Run quickly. • Allow for easy implementation. Workshop on Content-Based Access of Image and Video Libraries
f Proposal: Altered-Image Queries Retrieved ranks: Image Library 1 Look for rank of original: Query 2 3 Altered Image Image from Library etc. Workshop on Content-Based Access of Image and Video Libraries
The Crop Test • Crop image to k% of its original area. • Simulates close-up shot of same subject. Original Crop-50 Workshop on Content-Based Access of Image and Video Libraries
The Jumble Test • Shuffle tiles in image divided on an hk grid. • Simulates image with similar elements in a different arrangement. Jumble-44 Original Workshop on Content-Based Access of Image and Video Libraries
The Low-Con Test • Decrease contrast to k% of its original range. • Simulates altered lighting conditions and/or camera differences. Original Low-Con-80 Workshop on Content-Based Access of Image and Video Libraries
Typical results • Most retrievals are at low rank. • A few retrievals are at much higher rank. Median: 26 Mean 205 • Median and mean summarize the results of • multiple repetitions. Workshop on Content-Based Access of Image and Video Libraries
Difficulty of Altered-Image Queries • Both mean and median increase with difficulty. • Note order-of-magnitude changes. Workshop on Content-Based Access of Image and Video Libraries
How Stable are Altered-Image Queries? • Ran Crop-50 on three entirely different sets of 6000 images. • Some consistency even with different test sets. • Look for order-of-magnitude change. Workshop on Content-Based Access of Image and Video Libraries
Does the Number of Images Matter? • Found linear dependence on number of images. Workshop on Content-Based Access of Image and Video Libraries
How Many Queries Must Be Run? • Small % of total image set gives decent figure. Workshop on Content-Based Access of Image and Video Libraries
Comparing Algorithms Using Altered-Image Queries • Three algorithms compared using altered image queries. • Especially good or bad performance can be identified. Workshop on Content-Based Access of Image and Video Libraries
Final Thoughts • Altered-Image Queries are... • Well defined. • Easy to implement. • Consistent over different image sets. • A useful addition to our evaluation toolkit. • Also offer diagnostic potential. Workshop on Content-Based Access of Image and Video Libraries