Outline

1. Outline • Content-Based Image Retrieval • Query-by-Example • Query-by-Feature • Feature Vector

2. CBIR and CBR • Content-based Image Retrieval (CBIR) as example of Content-based Retrieval (CBR) • concentrates on low-level features. • Main Ideas of CBIR: • Represent an image as a set of feature descriptors. • Define similarity measures of the descriptors. • When a user specifies a query, the system returns images, which are sorted by similarity.

3. CBIR Architecture

4. CBIR of Butterflies • To allow non-expert users to find out some possible species of the butterflies they saw by the appearance of the butterflies • The appearance: • Color, Texture, Shape

5. Problems • How can you describe a butterfly? • How can you communicate with a machine?

6. Problems • Different users have different perception. • Users may not remember the appearance of the butterfly clearly. • Users usually do not have enough knowledge to describe the butterflies like experts. • Users usually do not have patience to browse too many query results.

7. Solutions • An user-driven interactive query process: QBF/QBE query process • Query By Features and Query By Example • Fuzzy feature description for each butterfly • An “What You See Is What You Get” query interface • A representative set for our collection of butterflies

8. QBF/QBE query process (1) • QBF query: • A QBF query is to choose some features of butterflies and expect that the system returns all butterflies with those features. • Features of butterflies: • Dominant color, texture pattern, shape. • QBE query: • A QBE query is to point an image and expect that the system returns all butterflies similar to that.

9. QBF/QBE query process (2) • Properties of QBF: • Rough search • When to use: • The first query and when users want to enlarge the view in the search space • Properties of QBE: • Fine search • When to use: • Usually the last query and when users want to see the neighbors of the query one in the search space.

10. QBF/QBE query process (3) • Result page: (Each result page should contain two parts) • Result Images: • These are the butterfly images satisfy the query conditions. • Users can invoke QBE queries from these images. • Related Features: • These are the features related to the previous query conditions. • Users can invoke QBF queries from these features.

11. Feature Description (1) • Feature Description for a butterfly: • Like metadata which describe the appearance of this butterfly. • This makes QBF queries possible. • Feature Description consists of some feature descriptors. • Feature descriptor: • A ( “feature value” , “match level” ) pair.

12. Feature Description (2)

13. Feature Description (3) Color

14. Feature Description (4) Texture

15. Feature Description (5) Shape

16. Feature Description (6) • QBF query: • Single feature query: • Result images: images with its corresponding degree of match > 0. • Ranked by: degree of match. • We call this ranked sequence “Feature sequence.” • Multiple features query: • Merge the corresponding feature sequences.

17. Result Presentation • For QBF query: • Property: rough search • Presentation: representative butterflies only • For QBE query: • Property: fine search • Presentation: • For very similar images: present them all • For less similar images: representative ones

18. Feature Vector Indexing • Goal: • To make search efficiently. • Problems of Indexing in CBIR: • Dimension of feature space is very high. • Index structure should support Euclidean and non-Euclidean similarity measures. • Solution: • Dimension reduction: KLT, DCT, DWT. • Similarity indexing: R*-tree, SS-tree, SR-tree.

19. Semi-Automatic Feature Extraction • Segmentation: • Background segmentation • Butterfly object segmentation • Feature extraction: • Color: color histogram • Texture: manual annotation • Shape: manual annotation

20. Classic CBIR with Color Feature • Most of the CBR systems rely on the notion of color, this may differ: • Dominant color • Scalable color based on color histograms (local for one region, global for the whole image) • Color Structure Descriptor (incoporates the spatial structure) • Let’s see the CBIR color theory and describe how MPEG-7 copes with it. • Let’s start why we focus on color and 90% of the CBIR is on color.

21. What color is the apple ? We are so visual !!!! I’d say it is Bright Red I really couldn’t tell you (I am color blind) I think it is “Crimson” It is Red!

22. Cosmic rays X-rays UV IR Radio waves 400 nm-700 nm “visible light” Visible light is part of a wide electro-magnetic spectrum 100 1.0 0.8 0.6 0.4 0.2 0.0 Relative Sensitivity 0 400 500 600 700 200 400 600 800 1000 Wavelength (nm) Wavelength (nm) • Response of human eye (luminance response) • Response of Silicon

23. RGB Color Space • Hardware Oriented Model: RGB Color Space: 3 values to represent a color. Red Green Blue

24. HSV Color Space • HSV – Hue Saturation Value • close to human perception • 3 values to represent a color. Value Green (120o) Yellow (60o) Cyan (180o) Red (0o) Blue (240o) Magenta (300o) White Black Hue Saturation

25. YCbCr Color Space • Y is the Luminance and Cb and Cr are the Chrominance Values of this Color Space. • Decouples intensity and color information • A monochrome color representation has only the Y value. • Very Perceptual Model, see next slide.

26. Luminance and Chrominance Achromatic + Red cone + + (Luminance) + Rods + Red-Green - + Green Cone + (Chrominance) Yellow - Blue-Yellow + Blue cone (Chrominance) • Eye has higher sensitivity for Luminance

27. HMMD Color Space • HMMD (Hue-Max-Min-Diff) color space is closer to a perceptually uniform color space. • Min = min(R,G,B); • Max = max(R,G,B); • Diff = Max – Min; • Sum = (Max + Min)/2; • -Hue the same as in HSV

28. Max: indicates how much black color it has, giving a flavor of shade or blackness. Min: indicates how much white color it has, giving a flavor of tint or whiteness. Diff: indicates how much gray it contains and how close to the pure color, giving a flavor of tone or colorfulness. Sum: simulates the brightness of the color.

29. Color Space: Representation • RGB: [int, int, int] • Y Cb Cr: [int, int, int] • HSV: [int, int, int] • Value • lightness of the color • Saturation • how dominant a color is • Hue • dominant spectral tone of the color

30. v = max(R,G,B); s = (v-min(R,G,B))/v; if(R=max&&G=min) h=5+(R-B)/(R-G); else if(R=max&&B=min) h=1-(R-G)/(R-B); else if(G=max&&B=min) h=1+(G-R)/(G-B); else if(G=max&&R=min) h=3-(G-B)/(G-R); else if(B=max&&R=min) h=3+(B-G)/(B-R); else if(B=max&&G=min) h=5-(B-R)/(B-G);

31. Color Space: Distance • Weighted Euclidean Distance (L-2 metric) • Better Distance for HSV • w.r.t human perceptual system • interpreted as cylinder v h s

32. Distance Problem in Color Spaces: • a difference between green and greenish-yellow is relatively large, whereas the distance distinguishing blue and red is quite small. CIE 1931, chromaticity diagram

33. CIE-LAB Uniform Color Space • Uniform Color Spaces: CIE Lab • CIE solved this problem in 1976 with the development of the Lab color space.

34. Color Histogram: Representation • A list of Color-Percentage pairs: • Describe the colors and its percentages in an image.

35. Color Quantization • Indexed Colors • A jpg Image with 256-color components in each RGB channel • 256 x 256 x 256 colors in total → n groups, e.g, in 256 groups, that makes a reduction 256x256, I.e., that each group takes 256 colors to count.

36. In each group we compute how many pixels fall into this group, this gives e.g., 145. • Quantization means also the encoding of the bin values with a number of bits, • Criteria:

37. Color Representation in General T H Q B R b[m] S h[m] G V indexing B Linear or non-linear transformation Quantization Transform to binary

38. Example • Color space selection & quantization • Use RGB channels • Each channel is divided into 2 intervals • Total number of bins = 23 = 8

39. Example (cont.) • H(I): Color histogram for Image I • Three images of 8x8 pixels, each pixel in one of eight color bins C1 to C8 • Image 1 has 8 pixels in each eight color bins • H1 = (8, 8, 8, 8, 8, 8, 8, 8) • Image 2 has 7 pixels in each C1 to C4, and 9 pixels in each C5 to C8 • H2 = (7, 7, 7, 7, 9, 9, 9, 9)

40. Image 3 has 2 pixels in each C1 and C2, and 10 pixels in each C3 to C8 • H3 = (2, 2, 10, 10, 10, 10, 10, 10) • Quantization to 4 bins -> • H1 = (16, 16, 16, 16) • Quantization to 3bits -> • H1 = (7, 7, 7, 7)

41. Similarity Measures - Overview • Minkowski Similarity • Distance L1 : r = 1 • Distance L2 : r = 2 • Quadratic Similarity • Intersection Similarity (Swain et Ballard 1991)

42. Example (cont.) • Minkowski Similarity • Is a L-1 metric where Ik and Jk is the number of pixels in bin k for image I and J • Distance between above three images • D(H1, H2) = 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 8 • D(H1, H3) = 6 + 6 + 2 + 2 + 2 + 2 + 2 + 2 = 24 • D(H2, H3) = 5 + 5 + 3 + 3 + 1 + 1 + 1 + 1 = 23

43. Example (cont.) • Minkowski Similarity • Is a L-2 metric • Distance between above three images • D(H1, H2) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)1/2 = 2.8 • D(H1, H3) = (36 + 36 + 4 + 4 + 4 + 4 + 4 + 4)1/2 = 9.8 • D(H2, H3) = (25 + 25 + 9 + 9 + 1 + 1 + 1 + 1) 1/2 = 8.5

44. QBIC distance • Weighted Euclidean distance (QBIC) • Is a L-2 metric(?) distance between histogram H1 and H2: D = (H1 - H2)T A (H1 - H2) where A is a symmetric color similarity matrix A (i, j) = 1 –d (ci, cj) / dmax where ci and cj are the i-th and j-th color bins, d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space

45. Limitation • Ignore similarity between colors • Example • Two color bins • Bin-1 color range: 1 – 10 • Bin-2 color range: 11 – 20

46. Three color pixels • Pixel 1 is Color 10  Bin-1 • Pixel 2 is Color 11  Bin-2 • Pixel 3 is Color 20  Bin-2 • Pixel 2 is similar to Pixel 3 than Pixel 1  unreasonable !

47. Limitation (cont.) • Ignore spatial relationships among pixels Different image with same histogram

48. Spatial RelationshipsAmong Pixels • Segment image into regions & calculate histogram for each region • Remove background & calculate histogram of foreground • Correlogram • Use spatial layouts (see MPEG-7)

49. Color Representation in MPEG-7 MPEG-7 distinguishes: • Color Space and Color Quantization, • Descriptive Elements: • dominant, scalable, (histogram based, counting of pixels that fulfill criteria), • color-structure, layout, (spatial layout), • GoF/GoP (for video frames), • For both global image and regions of image.

50. Color Space Description in MPEG-7 Like : RGB, YCrCb, HSV, HMMD, linear transformation with reference to RGB