Visual Querying By Color Perceptive Regions

1. Visual Querying By Color Perceptive Regions Alberto del Bimbo, M. Mugnaini, P. Pala , and F. Turco University of Florence, Italy Pattern Recognition, 1998

2. Introduction • Relevance of visual elements depends on: • User’s subjectivity (not known in advance) • Context of application (known in advance) • Efficient system for retrieval by visual content: • Provide a query paradigm that allow users to naturally specify both selective and imprecise queries • Define retrieval facilities that are meaningful for the context of application • Define similarity metrics which are satisfactory for user perception

3. Introduction • PICASSO system allows image indexing and retrieval based on shapes, colors, and spatial relationships • This paper concentrates on color facilities • System supports both retrieval by global color similarity and retrieval by similarity of local color regions • Shape, size, and position of color regions are considered as optional features that the user can select in the query

4. Hierarchical Color Image Segmentation • Image segmentation: partition of an image into a set of non-overlapped regions - homogeneous with respect to some criteria - whose union covers the entire image • PICASSO: • Multiple descriptions of each image, each covers a different level of precision • Each database image is segmented into uniform color regions at different degrees of resolution, so as to obtain a pyramidalmulti-resolution representation

5. Hierarchical Color Image Segmentation • Minimize the associated energy: • By a heuristic approach, starting from the finest level of resolution: • Every pair of adjacent regions is checked to verify if their merge decreases the image energy; the two regions that provide the maximum decrease of image energy are merged • Continue the above process until a minimum of image energy is reached • Parameters are then changed so that a new minimum is reached with a lower number of regions

7. Hierarchical Color Image Segmentation • After the segmentation process, for an image, N segmented images are obtained and represented through a multi-layered graph • Region Vn.k is connected through intra-level links to neighboring regions, and through inter-level links to its son regions at layer n-1 • The graph is a multi-resolution index of the chromatic and positional content of the image

9. Color Region Representation • Use CIE L*u*v* color space • close distances in color space correspond to close distances in user’s perception • Computation of color distance between two generic points in L*u*v* space requires to evaluate the length of the shortest path linking the two points • Notpossible in real time because of its complexity

10. Color Region Representation • Uniform tessellation of L*u*v* color space • Number of colors reduced to a small set of reference colors • Distance between two generic colors belonging to the same reference color is well approximated by Euclidean distance • Experiments showed 128 colors suffice to achieve a reasonable compromise between accuracy and computational effort • Distances between reference colors are pre-computed

11. Region Description • At coarsest resolution: image is represented by a single region and color vector retains global color characteristics for the entire image • As resolution increases: regions correspond to smaller areas in image and therefore have a smaller number of reference colors • For a generic region Rn (at level n of resolution), with k-child regions (at level n-1), the color vector is computed as union of color vectors associated with child regions, hence

12. Region Description • Color regions are modeled through: • Area • Where #R = region and #I = image pixels • Spatial location • Absolute position of its centroid • Shape • Using the first 13 central moments defined as: • Binary 128-dimensional color vector

13. Color Image Retrieval • PICASSO supports retrieval by visual example of images with one or more colored regions • Queries are formulated through visual examples • Regions can be either sketched and then filled with appropriate colors or extracted from images • Similarity of color regions takes into account either chromatic qualities of sketched regions or combination of chromatic and spatial attributes • Highest node of each pyramid includes both the binary color vector associated with the whole image and the full image histogram

14. Color Image Retrieval • Color index file: stores the color vectors associated with the highest node of each image pyramid • For pruning unqualified images • Matching score (M) between a query region RQ and a database image region RI : • Similarity coefficient for the whole image:

15. Results • PICASSO allows: • Querying by color regions (positions are unimportant) • Querying by global color similarity (histogram) • Test for querying by color regions: • Features: color/area, color/position, color/position/area • Retrieved images were shown to (30) users who evaluate if retrieved images are relevant/ not relevant • Use data to compute precision and recall • Best values for color/position queries

17. Conclusion • System is used by Alinari Archives in Florence • Multi-resolution segmentation • Algorithms and index structure seem computationally complex