Integrating Color And Spatial Information for CBIR

1. Integrating Color And Spatial Information for CBIR NTUT CSIE D.W. Lin 2003.8.26

2. References • L. Cinque, G. Ciocca, S. Levialdi, A. Pellicano, and R. Schettini, “Color-based image retrieval using spatial-chromatic histograms,” Image and Vision Computing, 19 (2001) 979-986 • M.S. Kankanhalli, B.M. Mehtre, and H.Y. Huang, “Color and spatial feature for content-based image retrieval,” Pattern Recognition Letters, 20 (1999) 109-118 • S. Berretti, A.D. Bimbo, and E. Vicario, “Spatial arrangement of color in retrieval by visual similarity,” Pattern Recognition 35(2002) 1661-1674

3. The representation of color • Histogram refinement • Specific-color pixel distribution (single, pair, triple …) • Edge histogram … • Color histogram • Global color histogram • Global color histogram + spatial info. • (fixed) Partition + local color histogram • Dominant color • Extracting the representative colors of image via VQ or clustering (e.g. k-means algorithm) • Spatial info. can be attained Non-adaptive

4. Spatial-chromatic histograms [1] • SCH – global color histogram with info. about (single) pixel distribution • SCH attempts to answer: • How many meaningful colors?  color space quantization • Where the pixels having the same color?  location of region(with same color) • How are these pixels spatially arranged?  distribution of region

5. SCH – Color representation • Color representation • CIELAB  Munsell  ISCC-NBS • CIELAB • CIEXYZ CIELAB CIELUV • a, b: opponent color ( green  red, blue  yellow )

6. SCH – Color representation • Munsell color system • Hue value chroma

7. SCH – Color representation • ISCC-NBS Centroid Color System • Partitioning the Munsell color system into 267 blocks, each blocks represented by an unique linguistic tag and the block centroid (Munsell coordinates) • Using back-propagation NN to transform • CIELAB  Munsell  ISCC-NBS

8. SCH – Feature vector • The definition of SCH for image I SI(k) = (hI(k), bI(k), σI(k)) • k: kthquantized color (1~c) • hI(k): pixel amount(ratio) • bI(k): baricenter (normalized mean coordinates) • σI(k): standard deviation of (spread) • Properties: • Insensitive to scale changes(via normalization) • Compact representation and rapidly computing

9. SCH – Similarity measure • Similarity function • c: number of quantized color • d(·): Euclidean distance • : max. distance

10. SCH – Effectiveness measure • S: relevant items in DB • : retrieved set (short list) for a query • : relevant items in retrieved set

11. Color and spatial clustering [2] • k-means algorithm • Iteration version • Two-pass version • VQ (LBG algorithm) • Proposed color clustering (two-pass) • Generating a new cluster while d(p, Ci) > T • Merging those clusters with small population to the nearest cluster

12. Color and spatial clustering • Spatial clustering • Based on the clustered color layer • Using connected components labeling to separate the spatial clusters • Discarding those clusters with small population or lower density(embedded rectangular)

13. Feature vectors • For image I, color clusters can be given Cci = {Ri, Gi, Bi, λci, xci, yci} i: 1..m(number of color cluster) Ri, Gi, Bi:representative color of cluster λci: pixel ratio of cluster to total xci, yci : centroid of cluster fc={Cci|i=1, 2, …, m} • Do the same to color-spatial clusters

14. Similarity measure • 1: color distance between color cluster(RGB) • 2: relative frequency of pixels of color cluster () • 3: spatial distance between color cluster(x, y) • 4: relative frequency of pixels of color-spatial cluster () • 5: spatial distance between color-spatial cluster(x, y)

15. Spatial arrangement of color[3] • The back-projection from dominant colors to the image results in an exceedingly complex model(e.g. [2]) • Authors proposed a descriptor, called weighted walkthroughs, to capture the binary directional relationship of two complex sets of pixels

16. +,+ w+1+1 -, + -, - +, - a B Weighted walkthroughs • The model can be extended to represent the relationship of two sets A, B • w11 evaluates the number of pixel pairs aA and bB such that b is upper right from a Ci: characteristic function of negative/positive real number |B| : area of B i,j : ±1

17. Compositional computation • Reducing the region to a set of rectangular • The weight between A and B1B2 can be derived by linear combination of A/B1 and A/B2

18. +,+ w+1+1 -, + -, - +, - a B Distance of WW B • 3 directional indexes: • wH(A, B) = w1, 1 (A, B) + w1, -1 (A, B) • wV(A, B) = w-1, 1 (A, B) + w1, -1 (A, B) • wD(A, B) = w-1, -1 (A, B) + w1, 1 (A, B) • Spatial distance A B A B A

19. Arrangements comparing • Image model: < E, a, w > E: set of spatial entities (color-clustered region) a: E A { anya} (chromatic label) w: E E  W { anys} (spatial description)

20. Arrangements comparing • Distance between image model Q and D • : injective function(interpretation, association between query and model image) • DA : chromatic distance (L*u*v*) • DS : spatial distance • Nq : number of entities in query

21. Future works • Finish the color-spatial study(geometric-enhanced histogram) • Study wavelet and JPEG 2000