Image Retrieval: Current Techniques, Promising Directions, and Open Issues

Image Retrieval: Current Techniques, Promising Directions, and Open Issues Yong Rui, Thomas Huang and Shih-Fu Chang Published in the Journal of Visual Communication and Image Representation. Presented by: Deepak Bote

Presentation Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

History of Image Retrieval • Traditional text-based image search engines • Manual annotation of images • Use text-based retrieval methods • E.g. Water lilies Flowers in a pond <Its biological name>

Limitations of text-based approach • Problem of image annotation • Large volumes of databases • Valid only for one language – with image retrieval this limitation should not exist • Problem of human perception • Subjectivity of human perception • Too much responsibility on the end-user • Problem of deeper (abstract) needs • Queries that cannot be described at all, but tap into the visual features of images.

Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

What is CBIR? • Images have rich content. • This content can be extracted as various content features: • Mean color , Color Histogram etc… • Take the responsibility of forming the query away from the user. • Each image will now be described by its own features.

CBIR – A sample search query • User wants to search for, say, many rose images • He submits an existing rose picture as query. • He submits his own sketch of rose as query. • The system will extract image features for this query. • It will compare these features with that of other images in a database. • Relevant results will be displayed to the user.

Sample Query

Sample CBIR architecture

Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

Feature Extraction • What are image features? • Primitive features • Mean color (RGB) • Color Histogram • Semantic features • Color Layout, texture etc… • Domain specific features • Face recognition, fingerprint matching etc… General features

Mean Color • Pixel Color Information: R, G, B • Mean component (R,G or B)= Sum of that component for all pixels Number of pixels Pixel

Histogram • Frequency count of each individual color • Most commonly used color feature representation Corresponding histogram Image

Color Layout • Need for Color Layout • Global color features give too many false positives • How it works: • Divide whole image into sub-blocks • Extract features from each sub-block • Can we go one step further? • Divide into regions based on color feature concentration • This process is called segmentation.

Example: Color layout ** Image adapted from Smith and Chang : Single Color Extraction and Image Query

Texture • Texture – innate property of all surfaces • Clouds, trees, bricks, hair etc… • Refers to visual patterns of homogeneity • Does not result from presence of single color • Most accepted classification of textures based on psychology studies – Tamura representation • Coarseness • Contrast • Directionality • Linelikeness • Regularity • Roughness

Segmentation issues • Considered as a difficult problem • Not reliable • Segments regions, but not objects • Different requirements from segmentation: • Shape extraction: High Accuracy required • Layout features: Coarse segmentation may be enough

Problem of high dimensions • Mean Color = RGB = 3 dimensional vector • Color Histogram = 256 dimensions • Effective storage and speedy retrieval needed • Traditional data-structures not sufficient • R-trees, SR-Trees etc…

2-dimensional space Point A D2 D1

3-dimensional space

Now, imagine… • An N-dimensional box!! • We want to conduct a nearest neighbor query. • R-trees are designed for speedy retrieval of results for such purposes • Designed by Guttmann in 1984

IBM’s QBIC • QBIC – Query by Image Content • First commercial CBIR system. • Model system – influenced many others. • Uses color, texture, shape features • Text-based search can also be combined. • Uses R*-trees for indexing

QBIC – Search by color ** Images courtesy : Yong Rao

QBIC – Search by shape ** Images courtesy : Yong Rao

QBIC – Query by sketch ** Images courtesy : Yong Rao

Virage • Developed by Virage inc. • Like QBIC, supports queries based on color, layout, texture • Supports arbitrary combinations of these features with weights attached to each • This gives users more control over the search process

VisualSEEk • Research prototype – University of Columbia • Mainly different because it considers spatial relationships between objects. • Global features like mean color, color histogram can give many false positives • Matching spatial relationships between objects and visual features together result in a powerful search.

ISearch – my own system

Feature selection in ISearch

Database Admin facility in ISearch

Open issues • Gap between low level features and high-level concepts • Human in the loop – interactive systems • Retrieval speed – most research prototypes can handle only a few thousand images. • A reliable test-bed and measurement criterion, please!

Conclusion • Satisfactory progress, but still… A long way to go…!!

Acknowledgements • Dr. Padma Mundur • Mr. Yong Rao • Mr. Sumit Jain, Software Engineer, KPIT Cummins, India • Mr. Ajay Joglekar, Software Engineer, Veritas India.

References • Y. Rui, T. S. Huang, and S.-F. Chang, “Image retrieval: Current techniques, promising directions, and open issues” • S. Jain, A. Joglekar, and D. Bote, ISearch: A Content-based Image Retrieval (CBIR) Engine, as Bachelor of Computer Engineering final year thesis, Pune University, 2002

THANK YOU!!! Questions?

Image Retrieval: Current Techniques, Promising Directions, and Open Issues