Machine Learning on Images

1. Machine Learningon Images Janez Brank

2. Introduction • Collections of images(a.k.a. pictorial databases) • Image retrieval • Image classification • How do we represent images? • How do we classify them based onthis representation?

3. Global image descriptors (1) • Describe entire image with one vector • Histograms • Split (or “quantize”) the color spaceinto C disjoint regions. • For each color region, remember what percentage of the image is covered bypixels whose colors are from that region. • Color coherence vectors (Pass et al., 1996, 1999) • Distinguish pixels belonging to a largerpatch of pixels of the same color from the “lonelier” ones.

4. Global image descriptors (2) • Autocorrelograns (Huang et al., 1997) • Given a randomly chosen pixel of color c and a randomly chosen pixel at distance d from it,what is the probability that this second pixelalso has color c? • Store this for all c and for some d. • Banded autocorrelograms(Huang et al., 1998) • For each c, sum the correspondingacg entries over all d.

5. Segmentation • Divide the image into several “regions”. Each shouldbe uniform in terms of color and/or texture. • Describe each region separately. • How do we define similarity between imagesbased on the similarity between regions? • WALRUS (Natsev et al., 1998) • Slice the image into small windows or tiles (e.g. 44 pixels) • Describe a window using the principal coefficients ofthe wavelet transform ( a 12-dimensional vector) • Cluster the descriptions (and hence windows).

6. Similarity measures for segmented images • WALRUS: • Look for pairs of regions (one region fromeach image) whose descriptions are close enough. • Calculate the percentage of the imagescovered by these regions. • IRM (integrated region matching; Li et al., 2000) • Distance between images is expressed as aweighted sum of distances between regions • To assign weights, similarity between regions,as well as their size, is taken into account.

7. Machine learning techniques • The k nearest neighbor method: • Classify a new instance by finding the k training instancesclosest to it and choosing the class prevailing among them. • Support vector machines: • Instances assumed to be vectors; separate two classesby placing a hyperplane between them. • By pretending to map the vectors into a different space( kernel functions), we can achieve non-linear(e.g. polynomial) discrimination surfaces.

8. The database • 1172 images, chosen from a larger collection, classified manually into 14 disjoint classes

9. The experiments • Global image descriptors: • histograms, coherence vectors, autocorrelograms,banded autocorrelograms • Several color space quantizations:RGB (64, 216, 256 regions), HSV (256 regions) • Nearest neighbors (using L1- ali L2-norm asthe distance measure) • Support vector machines (three different kernels) • Segmentation: • Various segmentation parameters • Two similarity measures: WALRUS in IRM • Nearest neighbors

10. Conclusions • Autocorrelograms better than histograms, banded acg as good as full ones • The quantization of the color space should not be too rough

11. Conclusions • Autocorrelograms better than histograms, banded acg as good as full ones • The quantization of the color space should not be too rough • Support vector machines producebetter classifiers than thenearest-neighbor method

12. Conclusions • Autocorrelograms better than histograms, banded acg as good as full ones • The quantization of the color space should not be too rough • Support vector machines producebetter classifiers than thenearest-neighbor method • Cubic and radial kernels better than linear

13. Conclusions • Autocorrelograms better than histograms, banded acg as good as full ones • The quantization of the color space should not be too rough • Support vector machines producebetter classifiers than thenearest-neighbor method • Cubic, radial kernels better than linear • Segmentation parameters are important • IRM is better than the WALRUSsimilarity measure • Segmentation is not better thanglobal descriptors

14. Conclusions • Autocorrelograms better than histograms, banded acg as good as full ones • The quantization of the color space should not be too rough • Support vector machines producebetter classifiers than thenearest-neighbor method • Cubic, radial kernels better than linear • Segmentation parameters are important • IRM is better than the WALRUSsimilarity measure • Segmentation is not better thanglobal descriptors • Looking at more nearestneigbors yields worse results

15. Comparison with related work • Huang et al. (1998) built a sort ofdecision tree over a set of images,and also compared it with NN • Banded autocorrelograms,SVD (for each node of the tree) • Eleven classes, 90 images per class • Classification accuracies: around 80 %

16. Possibilities of future work • Understanding the unexpected results • The poor performance ofsegmentation-based classifiers • Why is k-NN best when k = 1? • Compare with other techniques • Other segmentation methods • Other color spaces • Looking for new image representationmethods and new similarity measures