Kai Uwe Barthel FHTW Berlin, Treskowallee 8, 10313 Berlin, Germany, barthel@fhtw-berlin.de

1. Improved image retrieval using automatic image sorting and semi-automatic generation of image semantics = Query Image Result Images ... Internetimage search Small searchresult set plant Kai Uwe Barthel FHTW Berlin, Treskowallee 8, 10313 Berlin, Germany, barthel@fhtw-berlin.de Keyword query Display of unsorted images • Problem of Image Search Systems • Both keyword-based and content-based image retrieval systems are not capable of searching images according to the human high-level semantics of images. • Keyword-Based Image Search • Most Internet images have not been annotated manually. Keywords are taken from the file names or words from the context of the web page containing the image. • Conventional Internet Image Search System (Query: „Eiffel Tower“) • As the intention of the searching user is not known to the search system, there will be problems due to homonyms, family names, labels, etc.: • Content-Based Image Retrieval (CBIR) • CBIR systems rely on the assumption that images similar to a query image do also share similar features. • There is an important semantic gap between the low level features and human high level semantic concepts. • Even sophisticated CBIR systems cannot determine similarities between semantically similar images. • Relevance Feedback (RF) • is used improve the retrieval quality. Problems of RF: • Users do not like to give feedback, feedback will be incomplete. • The search system does not know why a feedback was given, which feature was the reason for the feedback. • How to Collect Semantic Information • Users do not like to give feedback. However, they will mark candidate images in order to improve the quality of the retrieved image search result. • By selecting a set of candidate images a user expresses the fact that according to his desired search result these images do share some common semantic meaning. • Even though the particular semantic relationship is not known to the system, this selection of candidate sets - when collected over many searches - can be used to semi-automatically model the semantic inter-image relationships. • Inter-image relationships are described with weighted links, that describe how often two particular images have been selected together as part of a candidate set. • Every time a user selects of two or more candidate images, the links of these images will be updated. • Semantic Filtering • Semantic filtering can be achieved by retrieving those images having the highest link weight. • Combining keyword search and CBIR • Related approaches to combine high-level semantic keyword-based metadata and low-level statistical metadata to improve image retrieval: • Image retrieval systems allowing the user to choose from either a semantic or a visual queryWei Wang et al. (2003): SemView: A Semantic-sensitive Distributed Image Retrieval System • Automatic image annotationImages with similar low-level metadata were assigned to the same keywords. Jia-Yu Pan et al., 2004: Graph-based Automatic Image Captioning Problematic with images from different sources! • Image Search Techniques combining semantic keywords and low-level features.By using relevance feedback techniques links are assigned between the images and the keywords. • Y. L. Lu et al., 2000: A unified framework for semantics and feature based relevance feedback in image retrieval systemsX. Zhou, T. Huang, 2002: Unifying Keywords and Visual Contents in Image Retrieval • Linking images to keywords does help to filter out not-suiting images, but cannot help to distinguish between homonyms or different types of images. • Proposed New Scheme • New image search scheme using both keywords and low-level visual metadata to semi-automatically generate semantic inter-image relationships. • CBIR techniques to visually sort retrieved images • The visually sorted arrangement allows to inspect more result images. • Within this larger set the user can quickly identify those images, which are good candidates for his desired search result. • New Idea: Images are not linked to keywords but images are linked to other semantically similar images. • Semantic relationships are learned exclusively from the human users’ interaction with the image search system – from the . • The proposed system can be used to search huge image sets more efficiently. • Overview of the Proposed System • Visual Sorting • Images are sorted using a self-organizing map (SOM). • Best sorting results using optimized features derived from the MPEG-7 Color-Layout descriptor: • Only strong variance DCT coefficients • Higher weight for chrominance coefficients • No quantization • Square torus-shaped Batch SOM • Very fast sorting due to incremental filters(200 images in less than 50 ms) • Candidate images will be used to visually filter (refine) the search result and to learn the semantic inter-image relationships. • Visual Filtering • Best visual filtering results were achieved using the multimodal neighborhood signature. Huge search result set Internetimage search Internetimage search Larger searchresult set Calculation of visual filter parameters Visual filter Visual sorting Calculation of semantic filter parameters Semantic filter 2a: Selection of result candidates Display of images sorted by visual similarity 1: Keyword query 2b: Query with the same keyword Refined result set Database Unsorted result images (in the order as delivered by Google) Visually sorted result images(candidate images marked in red) Refined result images (visually filtered from 840 images)