1 / 17

INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval

INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval. H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín, Manuel Montes , Eduardo Morales, Enrique Sucar, Luis Villaseñor Language Technologies Laboratory National Institute of Astrophysics, Optics and Electronics

LeeJohn
Download Presentation

INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. INAOE at ImageCLEF2007Towards Annotation based Image Retrieval H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín,Manuel Montes, Eduardo Morales, Enrique Sucar, Luis Villaseñor Language Technologies Laboratory National Institute of Astrophysics, Optics and Electronics Tonantzintla, Mexico mmontesg@inaoep.mx http://ccc.inaoep.mx/~mmontesg

  2. Overview of the talk • Our first participation at ImageCLEF; the goal was to build the basic infrastructure • Some textual and mixed strategies for image retrieval • However we could do something more… • A Web based query expansion method, and • An annotation based image retrieval approach

  3. Terms for annotations Example images CBIR Query QueryExpansion TBIR Topic statement RelevantImages Textual and mixed strategies • VSM IR System for textual retrieval (baseline) • Late fusion of independent retrievers (LF) • Intermedia feedback (IMFB) Topic statement TBIR Fusion Query RelevantImages CBIR Example images

  4. Some new things… • Web-based query expansion: • Original statement + top-k snippets (NQE) • Original statement + top-l more repeated words from the top-k snippets (WQE) • Annotation based expansion (ABE) • Use automatic image annotation methods for obtaining text from images, then… • Expand documents and/or queries with automatic annotations, finally… • Apply some strategy for textual image retrieval

  5. Basis of our idea • Region-level annotations are generally complementary to manual (image-level) annotations sky palm palm, sky, sand, grass, sea, clouds clouds Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil; sea sand sand grass Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil; Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;

  6. x1 x1 x1 x2 x2 x2 …….. …….. …….. xn xn xn Automatic image annotation • Assign labels (words) to regions within segmented images Automatic image Annotation method . . . Sky Elephant Grass 0.6 Sky 0.2 Tree 0.1 Ground 0.1 Annotation improvement Rock 0.5 Church 0.2 Elephant 0.2 Entrance 0.1 Grass 0.5 Tree 0.3 Ground 0.1 Jet 0.1 Grass

  7. R1R2R3R4 c1 grasspeopletreechurch c2 grass tree treechurch c.. ….….….…… ci rockpeopletree church c.. ….….….…… cj treepeopletree church c256 buildingjetjet elephant Idea: select the best label’s configuration, taking into account: 1. The prior probabilities of each label, and 2. The semantic cohesion of the entire configuration Improving the automatic annotation Grass 0.6 Tree 0.2 rock 0.1 building 0.1 People 0.4 Tree 0.3 Mountain 0.2 Jet 0.1 Tree 0.5 Grass 0.3 Sky 0.1 Jet 0.1 Grass, Tree, Rock, Building, People, Mountain, Jet, Sky, Church, Elephant Church 0.3 Grass 0.3 Sky 0.2 Elephant 0.2

  8. Set of labels

  9. Some problems with the labels • 2000 training annotated-regions (2%) • 98000 regions to annotate (98%) • Imbalanced training set • Limited vocabulary

  10. accommodation with swimming pool sky water tree sand boats sky tree people water sand sky tree buildings accommodation with swimming pool + sand boats sky tree people water buildings + three given images Annotation based query expansion

  11. Annotation based document expansion The surroundings of the Valle Francés Torres del Paine National Park, Chile March 2002 furniture grasspeople clouds The volcano Tungurahua Baños, EcuadorMarch 2002 sand clouds sky mountain

  12. Experimental results Top ranked runs for each configuration considered.

  13. Automatic annotations AutomaticAnnotation Exampleimages TBIR RelevantImages CBIR Terms for manual annotations Visual-English run • No textual query was used, but at the end the recovery was done based on textual data. • It combines intermedia feedback and our annotation based expansion technique.

  14. Textual vs. mixed strategies

  15. Initial conclusions • Intermedia feedback is an effective way for mixing visual and textual information • Methods based on web-query expansion showed better performance • Anotation based expansion is a promising way for expanding text using image’s visual content • Annotations can be useful for image retrieval, though several issues should be addressed

  16. Our current work • Work on the improvement of automatic image annotation methods • Investigate different (better) ways for measuring the semantic cohesion between labels and manual annotations • Use such semantic cohesion estimates for improving image retrieval from annotated collections

  17. Thanks for your attention Language Technologies Laboratory National Institute of Astrophysics, Optics and Electronics Tonantzintla, México Manuel Montes y Gómez mmontesg@inaoep.mx http://ccc.inaoep.mx/~mmontesg

More Related