Content-based Image Retrieval (CBIR)

1. Content-based Image Retrieval (CBIR) Searching a large database for images that match a query: • What kinds of databases? • What kinds of queries? • What constitutes a match? • How do we make such searches efficient?

2. Applications • Art Collections e.g. Museums, galleries, Archives • Medical Image Databases CT, MRI, Ultrasound, The Visible Human • Scientific Databases e.g. Earth Sciences, Hubble • General Image Collections for Licensing Corbis, Getty Images, photo galery of Gulhasan • The World Wide Web

3. What is a query? • an image you already have • a rough sketch you draw • a symbolic description of what you want e.g. an image of a man and a woman on a beach

4. Some Systems You Can Try Corbis Stock Photography and Pictures http://www.corbis.com/ • Corbis sells high-quality images for use in advertising, • marketing, illustrating, etc. • Search is entirely by keywords. • Human indexers look at each new image and enter keywords. • A thesaurus constructed from user queries is used.

5. AVAİLABLE CBIR SYSTEMS

6. QBIC IBM’s QBIC (Query by Image Content) http://wwwqbic.almaden.ibm.com • The first commercial system. • Uses or has-used color percentages, color layout, • texture, shape, location, and keywords.

7. Blobworld UC Berkeley’s Blobworld http://elib.cs.berkeley.edu/photos/blobworld • Images are segmented on color plus texture • User selects a region of the query image • System returns images with similar regions • Works really well for tigers and zebras

8. Ditto Ditto: See the Web http://www.ditto.com • Small company • Allows you to search for pictures from web pages

9. National Archives, USA Declaration of Independence

10. Ottoman Archives ???

11. QUERY BY EXAMPLE

12. Feature Space Images Image Features / Distance Measures Query Image Retrieved Images User Image Database Distance Measure Image Feature Extraction

13. Features • Color (histograms, gridded layout, wavelets) • Texture (Laws, co-occurence, Gabor filters, local binary partition) • Shape (first segment the image, then use statistical • or structural shape similarity measures) • Objects and their Relationships • This is the most powerful, but you have to be able to • recognize the objects!

14. Color Histograms

15. QBIC’s Histogram Similarity The QBIC color histogram distance is: dhist(I,Q) = (h(I) - h(Q)) A (h(I) - h(Q)) T • h(I) is a K-bin histogram of a database image • h(Q) is a K-bin histogram of the query image • A is a K x K similarity matrix

16. Similarity Matrix RGBYCV 1 0 0 .5 0 .5 0 1 0 .5 .5 0 0 0 1 1 1 1 R G B Y C V ? How similar is blue to cyan? ?

17. Gridded Color Gridded color distance is the sum of the color distances in each of the corresponding grid squares. 2 2 1 1 3 4 3 4 What color distance would you use for a pair of grid squares?

18. Texture Distances • Pick and Click (user clicks on a pixel and system • retrieves images that have in them a region with • similar texture to the region surrounding it. • Gridded (just like gridded color, but use texture). • Histogram-based (e.g. compare the LBP histograms).

19. Local Binary Pattern Measure • For each pixel p, create an 8-bit number b1 b2 b3 b4 b5 b6 b7 b8, • where bi = 0 if neighbor i has value less than or equal to p’s • value and 1 otherwise. • Represent the texture in the image (or a region) by the • histogram of these numbers. • Compute L1 distance between two histograms: 1 2 3 100 101 103 40 50 80 50 60 90 4 5 1 1 1 1 1 1 0 0 8 7 6

20. Laws Texture

21. Law’s texture masks (1)

22. Shape Distances • Shape goes one step further than color and texture. • It requires identification of regions to compare. • There have been many shape similarity measures • suggested for pattern recognition that can be used • to construct shape distance measures.

23. Global Shape Properties:Projection Matching 0 4 1 3 2 0 Feature Vector (0,4,1,3,2,0,0,4,3,2,1,0) 0 4 3 2 1 0 In projection matching, the horizontal and vertical projections form a histogram. What are the weaknesses of this method? strengths?

24. Global Shape Properties:Tangent-Angle Histograms 135 0 30 45 135 Is this feature invariant to starting point?

25. Boundary Matching:Fourier Descriptors • Given a sequence of points of a boundary Vk

26. Boundary Matching • Elastic Matching The distance between query shape and image shape has two components: 1. energy required to deform the query shape into one that best matches the image shape 2. a measure of how well the deformed query matches the image

27. Del Bimbo Elastic Shape Matching query retrieved images

28. Our work on CBIR at METU • M. Uysal, F.T. Yarman-Vural, “A Content-Based Fuzzy Image Database Based on The Fuzzy ARTMAP Architecture”, Turkish Journal of Electrical Engineering & Computer Sciences, vol. 13, pp.333-342, 2005. • Ö.Ö. Tola, N. Arıca, F.T. Yarman-Vural, “Shape Recognition with Generalized Beam Angle Statistics”, Lecture Notes on Computer Science, vol. 2, 2004. • Ö.C. Özcanli, F.T. Yarman-Vural, “An Image Retrieval System Based on Region Classification”, Lecture Notes on Computer Science, vol. 1, 2004. • Y.Xu, P.Duygulu, E.Saber, A.M. Tekalp, F.T. Yarman-Vural, “Object-based Image Labeling Through Learning by Example and Multi-level Segmentation”, Pattern Recognition, vol. 36, pp.1407-23, 2003. • N. Arıca, F.T. Yarman-Vural, “BAS: A Perceptural Shape Descriptor Based on the Beam Angle Statistics”, Pattern Recognition Letters, vol. 24, pp.1627-39, 2003. • M. Ozay, F.T. Yarman-Vural. " On the Performance of Stacked Generalization Architecture", Lecture Notes in Computer Science, ICIAR pp.445-451, 2008, • E. Akbaş, F.T. Yarman-Vural, "Automatic Image Annotation by Ensemble of Visual Descriptors", CVPR 2007,pp.1-8, 2007. • İ. Yalabık, F.T. Yarman-Vural, G. Ucoluk, O.T. Sehitoglu, "A Pattern Classification Approach for Boosting with Genetic Algorithms", ISCIS07, pp.1-6, 2007. • M. Uysal, E. Akbas, F.T. Yarman-Vural, “A Hierarchical Classification System Based on Adaptive Resonance Theory”, ICIP06, p.2913-16, 2006. • E. Akbaş, F.T. Yarman-Vural, “Design of Feature Set for Face Recognition Problem”, Lecture Notes in Computer Science, ISCIS 2006, pp.239-47, 2006. • N. Arica, F.T. Yarman-Vural, “Shape Similarity Measurement for Boundary Based Features”, Lecture Notes in Computer Science, ISBN 3-540-29069-9, 3656, pp.431-438, 2005. • M. Uysal and F.T. Yarman-Vural, “ORF-NT: An Object-Based Image Retrieval Framework Using Neighborhood Trees”, Lecture Notes in Computer Science 37332005, ISBN 3-540-29414-7 SCIS2005, p.595-605, 2005.

29. N. Arıca and F.T. Yarman-Vural, “A Compact Shape Descriptor Based on the Beam Angle Statistics”, International Conference on Image and Vidor Retrieval (CIVR), 2003. • N. Arıca, F.T. Yarman-Vural, “A Perceptual Shape Descriptor”, International Conference on Pattern Recognition ICPR, 2002. • A. Çarkacıoğlu, F.T. Yarman-Vural "Learning Similarity Space", Proc. Int. Conf. on Processing ICIP02, pp. 405-408, Rochester, NY, USA, September 2002. • N. Arıca, F.T. Yarman-Vural, “A Perceptual Shape Descriptor”, Proc. International Conference on Pattern Recognition(ICPR) Quebec, Canada, 2002. • N. Arıca, F.T. Yarman-Vural, “A Shape Descriptor Based on Circular Hidden Markov Model”, Proc. International Conference on Pattern Recognition(ICPR), 2000, Barcelona, Spain. • P. Duygulu, E. Saber, M. Tekalp, F.T. Yarman-Vural, “Multi-Level Image Segmentation for Content Based Image Representation" IEEE-ICASSP-2000, July 2000, Istanbul. • P. Duygulu, A. Çarkacıoğlu, F.T. Yarman-Vural, "Multi-Level Object Description: Color or Texture", First IEEE Balkan Conference on Signal Processing, Communications, Circuits, and Systems, June 1-3, 2000, Istanbul, Turkey. • N. Dicle, F.T. Yarman-Vural, NES A File Format for Content Based Coding of Images, ISCIS 15, 2000, Istanbul. • N. Arıca, F.T. Yarman-Vural, “A New HMM Topology for Shape Recognition” IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing (NSIP'99), pp. 162-168, Antalya TURKEY, 1999. • S. Genc, F.T. Yarman-Vural, "Morphing for Motion Estimation" Int.Conf. on Image Processing Systems, and Technologiesand Information Systems, Las Vegas 1999.

30. Türkçe Bildiriler  • M. Özay, F.T. Yarman Vural, "Yığılmış Genelleme Algoritmalarının Performans Analizi", Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2008. • A. Sayar, F.T. Yarman Vural, "Yarı Denetimli Kümeleme ile Görüntü İçeriğine Açıklayıcı Not Üretme", Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2008. • Ö. Özcanli, E. Akbaş, F.T. Yarman Vural, “Görüntü Erişiminde Bulanık ARTMAP Ve Adaboost Sınıflandırma Yöntemlerinin Performans Karşılaştırması”, IEEE 13. Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2005. • Ö.Ö. Tola, N. Arıca, F.T. Yarman Vural, “Genelleştirilmiş Kerteriz Açısı İstatistikleri ile Şekil Tanıma”, Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 1, 2004. • M. Uysal, Ö. Özcanlı, F.T. Yarman Vural, “Bulanık ARTMAP Mimarisini Kullanan İçerik Bazlı Bir İmge Sorgulama Sistemi, Sinyal Isleme ve İletisim Uygulamaları Kurultayı, 2004. • A. Çarkacıoğlu, F.T. Yarman Vural, "Doku Benzerliğini Tesbit Etmek İçin Yeni Bir Tanımlayıcı", 11. SIU, Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2003. • N. Arıca, F.T. Yarman Vural, "Tıkız Şekil Betimleyicileri", 11.SIU, Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2003. • Ö.C. Özcanlı, P. Duygulu Şahin, F.T. Yarman Vural, “Açıklamalı Görüntü Veritabanları Kullanarak Nesne Tanıma ve Erişimi”, 11.SIU, Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2003. • S. Alkan, F.T. Yarman Vural, “Öğretim Üyesi Yetiştirme Programı”, Elektrik Elektronik, Bilgisayar Mühendislikleri Eğitimi I. Ulusal Sempozyumu, 2003. • M. Uysal, F.T. Yarman Vural, “En İyi Temsil Eden Öznitelik Kullanılarak İçeriğe Dayalı Endeksleme ve Bulanık Mantığa Dayalı Sorgulama Sistemi”, 11.SIU, Sinyal İşleme ve İletişim Uygulamaları Kurultayı, 2003. • N. Arıca, F.T. Yarman Vural, “Kerteriz Tabanlı Şekil Tanımlayıcısı”, 10. SIU, cilt.1 sayfa 129-134, 2002. • Ö.C. Özcanlı, S. Dağlar, F.T. Yarman Vural, “Yerel Benzerlik Örüntüsü Yöntemi ile Görüntü Erişim Sistemi”, 10. SIU, cilt.1 sayfa 141-146, 2002.

31. BAS: Beam Angle Statstics Ömer Önder Tola Nafiz Arıca Fatoş Tünay Yarman Vural

32. Beam Angle Statistics (BAS) • BAS, is a shape descriptor that describes a shape that is defined as an ordered set of boundary pixels, by assigning the statistics of beam angles measured for the pixel as a feature vector. • Beam Angle shows how much the shape is bended. • BAS, describes the 2D shape as 1D moment functions of the Beam Angles.

33. Beam Angle Statistics: Given an ordered set of boundary pixels 2 1 3 36 4 35 5 34 33 6 7 32 8 31 9 30 10 29 11 28 12 13 27 14 26 15 25 16 17 24 22 21 20 19 18 23

34. Beam Angle Statistics 1 0 2 1 3 2 4 3 4 5 6 5 7 6 8 7 9 8 10 9 11 12 10 13 11 14 12 15 16 13 15 16 17 18 17 14

35. Beam Angle Statistics 1 0 2 1 3 2 4 3 4 5 6 5 7 6 8 7 9 8 10 9 11 12 10 13 11 14 12 15 16 13 15 16 17 18 17 14

36. Beam Angle Statistics 1 0 2 1 3 2 4 3 4 5 6 5 7 6 8 7 9 8 10 9 11 12 10 13 11 14 12 15 16 13 15 16 17 18 17 14

37. 1 0 2 1 3 2 4 3 4 5 6 5 7 6 8 7 9 8 10 9 11 12 10 13 11 14 12 15 16 13 15 16 17 18 17 14 Beam Angle Statistics

38. Beam Angle Statistics 1 0 2 1 3 2 4 3 4 5 6 5 7 6 8 7 9 8 10 9 11 12 10 13 11 14 12 15 16 13 15 16 17 18 17 14

39. Beam Angle Statistics

40. Beam Angle Statistics

41. k=N/40 Plots of CK(i)’s with fix K values k=N/10 k=N/4 k=N/40 : N/4 METU, Department of Computer Engineering

42. What is the most appropriatevalue for K which discriminates the shapes in large database and represents the shape information at all scale ? Answer: Find a representation which employs the information in CK(i) for all values of K. Output of a stochastic process at each point METU, Department of Computer Engineering

43. C(i)is a Random Variable of a stochastic process which generates the beam angles mth moment of random variable C(i) Each boundary point i is representedby the moments of C(i) METU, Department of Computer Engineering

44. First three moments of C(i)’s METU, Department of Computer Engineering

45. Correspondence of Visual Parts and Insensitivity to Affine Transformation METU, Department of Computer Engineering

46. Robustness to Polygonal Approximation Robustness to Noise METU, Department of Computer Engineering

47. Similarity Measurement Elastic Matching Algorithm • Similarity Measurement method • Application of dynamic programming • Minimize the distance between two patterns by allowing deformations on the patterns. • Cost of matching two items is calculated by Euclidean metric. • Robust to distortions • promises to approximate human ways of perceiving similarity METU, Department of Computer Engineering

48. TEST RESULT FOR MPEG 7 CE PART A-1 Robustness to Scaling METU, Department of Computer Engineering

49. TEST RESULT FOR MPEG 7 CE PART A-2 Robustness to Rotation METU, Department of Computer Engineering

50. TEST RESULT FOR MPEG 7 CE PART B Similarity-based Retrieval METU, Department of Computer Engineering