Outline

1. Outline • Content-Based Image Retrieval • Query-by-Example • Query-by-Feature • Feature Vector

2. CBIR and CBR • Content-based Image Retrieval (CBIR) como exemplo de Content-based Retrieval (CBR) • concentra em low-level features. • Principais idéias de CBIR: • Representar uma imagem como um conjunto de feature descriptors. • Definir medidas de similaridade dos descritores • Quando um usuário especificar uma query, o sistema retorna imagens, que são ordenadas por similaridade.

3. CBIR Architecture

4. Image Retrieval Database Images Query Image Image Database Feature Extraction Feature Extraction Select Compare Metadatabase Feature Vectors Query Result

5. CBIR de Butterflies • Permitir non-expert users encontrar algumas espécies de butterflies usando informações de aparência de butterflies • Aparência: • Color, Texture, Shape

6. Problemas • Como podemos descrever uma butterfly? • Como podemos comunicar nossa descrição para uma máquina?

7. Problemas • Usuários diferentes têm percepções diferentes. • Usuários podem não se lembrar claramente a aparêcia de uma butterfly. • Usuários normalmente não têm expertise para descrever butterflies. • Usuários normalmente não têm paciência para fazer o browse num grande conjunto resultado.

8. Soluções • Usar um processo de consulta interativo e direcionado ao usuário: QBF/QBE query process • Query By Features e Query By Example • Fuzzy feature description para cada butterfly • Uma “What You See Is What You Get” query interface • Um conjunto representativo de coleção butterflies

9. QBF/QBE query process (1) • QBF query: • A QBF query is to choose some features of butterflies and expect that the system returns all butterflies with those features. • Features of butterflies: • Dominant color, texture pattern, shape. • QBE query: • A QBE query is to point an image and expect that the system returns all butterflies similar to that.

10. QBF/QBE query process (2) • Properties of QBF: • Rough search • When to use: • The first query and when users want to enlarge the view in the search space • Properties of QBE: • Fine search • When to use: • Usually the last query and when users want to see the neighbors of the query one in the search space.

11. QBF/QBE query process (3) • Result page: (Each result page should contain two parts) • Result Images: • These are the butterfly images satisfy the query conditions. • Users can invoke QBE queries from these images. • Related Features: • These are the features related to the previous query conditions. • Users can invoke QBF queries from these features.

12. Feature Description (1) • Feature Description for a butterfly: • Like metadata which describe the appearance of this butterfly. • This makes QBF queries possible. • Feature Description consists of some feature descriptors. • Feature descriptor: • A ( “feature value” , “match level” ) pair.

13. Feature Description (2)

14. Feature Description (3) Color

15. Feature Description (4) Texture

16. Feature Description (5) Shape

17. Feature Description (6) • QBF query: • Single feature query: • Result images: images with its corresponding degree of match > 0. • Ranked by: degree of match. • We call this ranked sequence “Feature sequence.” • Multiple features query: • Merge the corresponding feature sequences.

18. Result Presentation • For QBF query: • Property: rough search • Presentation: representative butterflies only • For QBE query: • Property: fine search • Presentation: • For very similar images: present them all • For less similar images: representative ones

19. Feature Vector Indexing • Goal: • To make search efficiently. • Problems of Indexing in CBIR: • Dimension of feature space is very high. • Index structure should support Euclidean and non-Euclidean similarity measures. • Solution: • Dimension reduction: KLT, DCT, DWT. • Similarity indexing: R*-tree, SS-tree, SR-tree.

20. Semi-Automatic Feature Extraction • Segmentation: • Background segmentation • Butterfly object segmentation • Feature extraction: • Color: color histogram • Texture: manual annotation • Shape: manual annotation

21. Classic CBIR with Color Feature • Most of the CBR systems rely on the notion of color, this may differ: • Dominant color • Scalable color based on color histograms (local for one region, global for the whole image) • Color Structure Descriptor (incoporates the spatial structure)

22. What color is the apple ? We are so visual !!!! I’d say it is Bright Red I really couldn’t tell you (I am color blind) I think it is “Crimson” It is Red!

23. Color Histogram: Representation • A list of Color-Percentage pairs: • Describe the colors and its percentages in an image.

24. Color Quantization • Indexed Colors • A jpg Image with 256-color components in each RGB channel • 256 x 256 x 256 colors in total → n groups, e.g, in 256 groups, that makes a reduction 256x256, I.e., that each group takes 256 colors to count.

25. Similarity Measures - Overview • Minkowski Similarity • Distance L1 : r = 1 • Distance L2 : r = 2 • Quadratic Similarity • Intersection Similarity (Swain et Ballard 1991)

26. Example (cont.) • Minkowski Similarity • Is a L-1 metric where Ik and Jk is the number of pixels in bin k for image I and J • Distance between above three images • D(H1, H2) = 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 8 • D(H1, H3) = 6 + 6 + 2 + 2 + 2 + 2 + 2 + 2 = 24 • D(H2, H3) = 5 + 5 + 3 + 3 + 1 + 1 + 1 + 1 = 23

27. Example (cont.) • Minkowski Similarity • Is a L-2 metric • Distance between above three images • D(H1, H2) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)1/2 = 2.8 • D(H1, H3) = (36 + 36 + 4 + 4 + 4 + 4 + 4 + 4)1/2 = 9.8 • D(H2, H3) = (25 + 25 + 9 + 9 + 1 + 1 + 1 + 1) 1/2 = 8.5

28. QBIC distance • Weighted Euclidean distance (QBIC) • Is a L-2 metric(?) distance between histogram H1 and H2: D = (H1 - H2)T A (H1 - H2) where A is a symmetric color similarity matrix A (i, j) = 1 –d (ci, cj) / dmax where ci and cj are the i-th and j-th color bins, d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space

29. Limitation • Ignore similarity between colors • Example • Two color bins • Bin-1 color range: 1 – 10 • Bin-2 color range: 11 – 20

30. Three color pixels • Pixel 1 is Color 10  Bin-1 • Pixel 2 is Color 11  Bin-2 • Pixel 3 is Color 20  Bin-2 • Pixel 2 is similar to Pixel 3 than Pixel 1  unreasonable !

31. Limitation (cont.) • Ignore spatial relationships among pixels Different image with same histogram

32. Noise-Free Queries (NFQ’s) • NFQ is more precise. • User can specify semantic constraints: • Spatial constraints (relative distances) • Scaling constraints (relative sizes) Rectangular query Noise-free query Similar Less relevant

33. Challenges • How do we extract features if we do not know the matching areas beforehand ? • How do we index the images ? Noise-free query

34. One Solution – Local Color Histogram (LCH) • Each subimage has a color histogram. • Any combination of the histograms can be selected for comparison with the corresponding color histograms of the query image.

35. Limitations of LCH • Dilemma: • Using large partitions is not precise • Using small partitions is too expensive • Limitation: • difficult to handle scaling

36. Resultados esperados de uma boa CBIR com segmentação Query 4 2 3 5 12 18 Query 3 216 396 2

37. DEMOS • Hermitage Museum Web Site (QBIC) http://hermitagemuseum.org/ http://hermitagemuseum.org/fcgi-bin/db2www/qbicColor.mac/qbic?selLang=English • http://www.aa-lab.cs.uu.nl/cbirsurvey/cbir-survey/