Supplementary Slides

1. Supplementary Slides

2. More experimental results • MPHSM already push out many irrelevant images Query image QHDM result, 4 of 36 ground truth found ANMRR=0.6464 MPHSM result, 9 of 36 ground truth found ANMRR=0.4819

3. More about experimental results • Still some irrelevant image found • No spatial information • Cannot identify background colors • Does not account for unmatched colors • Initial query might not be accurate Black Background Green Background

4. More about experimental results • Can be improved by • Relevance Feedback • Makes relevant images to have higher ranks • Irrelevant normally can’t have higher similarity by RF • But relevant images does • Give more information about the interested objects • Inconsistent backgrounds can be averaged out

5. More on experimental results • Irrelevant images got lower rank / out of top 20 after RF A E Query image B D First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688 Ground truth images A B C D A G E F E B D Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043 Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688

6. More about experimental results • Still some irrelevant images found • Some colors are very common (Blue sky, black night, green grass, etc.) • Different semantics might have similar color distribution • No single feature can do perfect retrieval • Can be improved by several approaches • Choose suitable features • Combining features

7. Suggestions on further developments • For DCD • Use unmatched colors • Challenge 1: Did the unmatched colors representing object of interest? Or just a obstacle? • Challenge 2: How to define the similarity function? • Separate foreground/background • Challenge 1: Can we identify it by only using DCD? Or in RF? • Challenge 2: Or we need to combine other shape/texture descriptors? • The DCD generation is not very accurate • GLA generates an optimal for quantizing the image, it might not be accurate dominant colors. • Can quantize up to 16 or more colors, and then approximate the least significant colors to obtain an 8 color DCD

8. Suggestions on further developments • For general CBIR • No single descriptor gives perfect retrieval • Choosing suitable features • Combining features (color+shape, color+texture, etc.) • Automatically? Manually? How to set weights?

9. Online Process User initial input Feature extraction Similarity measure Results output Similarity= 100% = 50% = 30% ... Similarity= 50% … Feature extraction Offline Process … Image DB Stored Features … … Visual description about a CBIR • System flow of a CBIR system

10. Color based CBIR approaches • Three major approach of CBIR based on colors • Area of matching – Count the area with matched colors (CSD, SCD, DCD) • Color distance – Use color distance to adjust the similarity (DCD-QHDM) • Spatial distribution – Matches colors having similar layout (CLD)

11. Optional parameters • Spatial coherence • obtained by a simple connected component analysis. A smooth surface gives a higher spatial coherence value. • Color variances • computed as variances of the pixel values within each cluster. But this parameter is for a dedicated similarity measure algorithm. So it is not commonly used.

12. Spatial coherency adjustment • Similarity measure • MPEG-7 suggests to use a modified Quadratic Histogram Distance Measure (QHDM) to measure the dissimilarity between descriptors • Spatial coherency adjustment

13. Results with Spatial Coherence • MPHSM improves DCD for both datasets to be more close to other non-compact descriptors • While using Corel_1k dataset MPHSM outperforms CLD slightly • MPHSM benefits from spatial coherency adjustment as well as QHDM

14. DCD-QHDM upper bound problem • Analysis of problem 1 • The upper-bound of the distance measured varies by number of color in the descriptor • Maximum of positive part is not a constant • Maximum of negative part is zero • So, the maximum of QHDM is not fixed • This property makes DCD unable to identify completely different images by the values measured Positive part Negative part

15. Upper bound problem - example • Problem 1 – The upper bound problem • Consider the following images with their DCD • I1, I2 are visually more similar than I1, I3 • For a similarity measure that matches human perception, we can expect the distance between F1, F2 should be smaller than that of between F1, F3 I I I 2 3 1 1/2 1/2 F F F 1/3 2 1 3

16. Upper bound problem - example • But distance between F1,F3 is smaller while measuring their distance using QHDM • The extra blue color pull down the distance • D2(F1,F2)>D2(F1,F3) implies that I1 is more similar to I3 than I2 • This shows that QHDM does not meet human perception

17. T d d a = 1.2 44% similar 16.67% similar 0% similar DCD-QHDM Similarity coefficient problem • The similarity coefficient use the color distance to fine tune the similarity • Difficult to define a quantitative similarity between colors, since the sensitivity of human eye depends on many conditions (e.g. light source of the room, spatial layout of the image, etc.)

18. Similarity coefficient problem • It is easy to count 50% of area is similar.But it is difficult to count the colors are 50% similar. • This method is unable to consider the area of matching and the color distance together.

19. I I I 2 1 4 1 1/2 1/2 F F F 2 1 4 Similarity coefficient problem - example • Problem 2 – The similarity coefficient a1i,2j problem • Consider the following images • I1, I2 are visually more similar than I1, I4 • For a similarity measure that matches human perception, we can expect the distance between F1, F2 should be smaller than that of between F1, F4

20. Similarity coefficient problem - example • But distance between F1,F4 is smaller while measuring their distance using QHDM • One exactly matched color considered more important than a whole area of similar color • D2(F1,F2)>D2(F1,F4) implies that I1 is more similar to I4 than I2 • But in natural perception, images having similar color distribution is more likely to have similar semantics • This shows that QHDM does not meet human perception again

21. Flow of MPHSM Initial DCDs Find a pair of colors with minimum distance d d<Td ? Merge colors having minimum distance Y N Common Palette Update each DCD based on the common palette Histogram Intersection

22. Merge colors Merge colors Merge colors Merge colors Palette Merging process, visually • Example • Two images with DCD, palette merging stage Find the closest pair Common Palette Remaining colors If a remaining color is similar to any colors in the common palette. It will not included in common palette Dominant Color Descriptor

23. About slide 23 • Relationship between CBIR and Relevance Feedback (RF) • The key component is query update

24. MPH-RF flow Load add DCDs A Append all DCD Cut least significant colors Find closes pair of colors Merge colors and percentages Adjust histogram sum into 1 Minimum distance < Td ? Y Updated query N A

25. RF of other MPEG-7 visual descriptors • Relevance feedback for MPEG-7 descriptors • Apart from the MPH-RF for DCD, we directly apply feature weighting technique on several MPEG-7 visual descriptors • RF on CLD: • RF on CSD:

26. RF of other MPEG-7 visual descriptors • RF on SCD:

27. MIRROR – A CBIR system using MPEG-7 visual descriptors • A set of visual descriptors • Relevance feedback functions is added • Evaluation tools • MIRROR is also a development platform of MPEG-7 visual descriptors

28. Performance of color descriptors • Performance of color descriptors • Evaluation tools • Unmodified MPEG-7 reference software XM • MPEG-7 Common Color Dataset (MPEG-7 CCD) with 5466 images and 50 sample queries • Corel 1000 images dataset with 20 sample queries • ANMRR performance metric (smaller means better)

29. Performance of color descriptors • Investigation of performances • Color structure descriptor performs best among color descriptors due to its large descriptor size • Dominant color descriptor performs worst, even worse than a more compact color layout descriptor • “Area of matching” is still the most efficient approach for color based CBIR • New methods will be proposed in this research to boost DCD

30. Complete results • MPHSM improves DCD for both datasets to be more close to other non-compact descriptors • While using Corel_1k dataset MPHSM outperforms CLD slightly • MPHSM benefits from spatial coherency adjustment as well as QHDM

31. Complete results • MPR-RF gives significant improvement on all combinations of similarity measures and datasets. • By using MPH-RF DCD can perform as good as another compact descriptor CLD, and very close to a lesser compact descriptor SCD. • Three iterations of relevance feedback give a significant result

32. Complete results • The MPH-RF improvement on DCD is more significant than feature weighting for other color descriptors • Color structure descriptor gives impressive results among all color descriptor, and its only drawback is the descriptor size is too large.