Survey of Algorithms to Query Image Databases

1. Image from Kodak’s PhotoQuilt <- Survey of Algorithms to Query Image Databases COMP 290-72:Computational Geometry Benjamin Lok 11/2/98

2. Outline of Talk • Overview of the problem • Three methods • Color based • Shape based • Vision based • Conclusions Image from Microsoft Clip Gallery

3. Problem • Query an image database • What does a “match” mean? • Application dependent • Notion of subjectivity • Sensitivity to noise

4. Problem • Semantic similarity is still not possible • ex. “All images with cats” • To determine similarity, we need to a new: • Metric • Space Images from Microsoft Clip Gallery and website

5. Images from “Shrine to Long Haired Men” and “Videos of Women Getting Their Heads Shaved” websites Ambiguity Girl Guy Guy

6. The Earth Mover’s Distance, Multi-Dimensional Scaling, and Color-Based Image Retrieval Yossi Rubner, Leonidas Guibas, Carlos Tomasi (1997) Stanford Vision Laboratory Image from Microsoft Clip Gallery

7. Color Signatures • Utilize the CIE-LAB color space • Based on human perception of color • Map each pixel to a point in color space • Common color values increase weight of point • Group clusters into points (8-12 per image) Rubner, Guibas, and Tomasi

8. Earth Mover’s Distance • To compare two images, compute the “work” needed to move the cluster points from one image to the other Rubner, Guibas, and Tomasi

9. Earth Mover’s Distance (cont) • Solving a linear programming problem: Given two signatures: p = {(p1,wp1),…,(pm,wpm)} and q = {(q1,wq1),…,(qn,wqn)} Find C where Cij is the amount of weight pi matched qj

10. Applications • Visualize Databases (Queries and Results) • Scale the multiple dimensions into 2D using MDS and minimize STRESS Rubner, Guibas, and Tomasi

11. Database Visualization Rubner, Guibas, and Tomasi

12. Algorithm Recap Image from YenPen Stationary Website • Map pixels to 3D color space points • Locate and compress “clusters” of points • 8 to 12 points determine the color signature • Calculate the Earth Mover’s Distance to determine “distance” between two images

13. Advantages Disadvantages • Based on human perception of color • Some invariance to small change in viewpoint and lighting • Meaningful metric • Relatively fast • Can embed multiple metrics • Depending on application, query format might be not be intuitive • Not much use for non-color images • False positives a real possibility depending on working domain

14. Image from Microsoft Clip Gallery Shaped-based Image Retrieval Using Geometric Hashing Scott D. Cohen and Leonidas J. Guibas 1997 Stanford Vision Laboratory

15. Overview • Implementation • Search through 500 Chinese characters • Goals • Provide invariance to scale, rotation, and translation • speed and accuracy Cohen and Guibas

16. Generating a Illustration • Illustration - set of curves that summarize an image • Edgel detection • Medial Axis determination Cohen and Guibas

17. Approximating with Polylines • Convert medial axis representation to polylines • Tradeoff between speed and accuracy Cohen and Guibas

18. Geometric Hashing Cohen and Guibas • Geometric Hashing - method used to compare two point sets under some transformation group • Take each point and use it as the origin of a coordinate system

19. Geometric Hashing (cont) Cohen and Guibas • If translating P by qj - pi produces a good match Ii(P) and Ij(Q) will match. • This property can be generalized to other transformation groups. • Each line segment is a basis of a coordinate system • Translation, Rotation, and Orientation defined • I(P) = transform all other segments into new CS

20. Notes on GH Cohen and Guibas • Each segment will be transformed to 2m Coordinate systems • I(P) stores O(m2) segments • Can be done as preprocessing step • Expressing the different possible transformations using each segment as a basis

21. Querying the Database • Query image undergoes the feature extraction process • For each query feature, a nearest-neighbor query is applied and the k closest or within some j • Similarity score increases if database image has a feature that is “close” to the query feature Cohen and Guibas

22. “Closeness” • How do you describe the closeness of two lines? • Transform to a 4D space made of (l,,a,b) • With two (l,,a,b) descriptions for lines, can compute distance • Divide by standard deviation over sample of database features

23. Details • Closeness is relative to database contents • Nearest-neighbor algorithm by Arya, Mount, et. al (1994). Query time for k nearest features is O(k log n) Cohen and Guibas

24. Advantages Disadvantages • Fast • Queries database of 500 characters in 1 second on SGI Indy • Queries based on important features • Working domain currently limited • Could get too expensive as complexity in images increases Cohen and Guibas

25. A Multi-Resolution Technique for Comparing Images Using the Hausdorff Distance Daniel Huttenlocher and William Rucklidge 1992 Cornell University Huttenlocher and Rucklidge

26. Directed Hausdorff Distance • Given A={a1, … , am} and B={b1, … , bm} • Identifies the point in A farthest from any point in B • Measures the degree of mismatch between between two sets.

27. Properties of Hausdorff Distances • Not symmetric h(A,B) != h(B,A) • Compute kth maximum • Notion of rank • Reduces sensitivity • Fraction of A within h(A,B) of B • Obscured portions • h(A,B) = hypothesis h(B,A) = test

28. Transformations t( ) = • Given A is an image, B is the model • Without Orientation, if A is in B then A undergoes transformation t. fB(t)=H(t(B),A) t=(tx,ty,sx,sy) forward fA(t)=H(A,t(B)) reverse

29. Bidirectional Hausdorff Distance • Solve for which values of t, the following holds: • Results in searching a four dimensional space

30. Restricting Search Space • Slope of f(t)=HLK(A,t(B)) is linear • Divide space into cells • Calculate HLK(A,tc(B)) • Determine a maximum delta per cell • Based on limit in scale and translation • Allows for quick rejection and acceptance • Label cells as interesting or disregard

31. Restricting Search Space • Create smaller cells from interesting cells • Bounds based on transformations • Quickly narrow down to areas that could possibly be within  of A

32. Subtleties • Discretization useful if working in computer vision domain (integers) • Can compare partially obscured images • Optimizations • Early rejection/acceptance • Pretty slow 200 to 250 seconds Website on submarines

33. Huttenlocher and Rucklidge

34. Advantages Disadvantages • Accurate • Geared towards image processing and vision • Partially obscured images • Searches similar to humans • Slow • No Orientation • Database must be specialized • Potential problems in generating queries

35. Recap • Three Algorithms • Color Based • Color Signatures • Earth Mover’s Distance • Shaped Based • Polylines • Transform Invariant Sets • Vision Based • Hausdorff Distance • Subdivision of Transformation Space www.sportsmanscaps.com

36. Final Thoughts www.jerryspringer.com • Algorithms work well in various domains • Query construction not formalized • Other methods: • wavelet-based • texture-based • object-based • Took 5 minutes to find “Shrine to Men with Long Hair” and “Videos of Women Getting Their Head Shaved” All other images generated by author using Paint Shop Pro