What Are the High-Level Concepts with Small Semantic Gaps?

1. What Are the High-Level Concepts with Small Semantic Gaps? CS 4763 Multimedia System, Spring 2008

2. Outline • Introduction • LCSS: a lexicon of high-level concepts with small semantic gaps. • Framework of LCSS Construction • Methodologies • Experimental results • Conclusion

3. Introduction • Recent years have witnessed a fast development of Multimedia Information Retrieval (MIR) . • Semantic gap is still a fundamental barrier. --Difference between the expressing power or descriptions of low-level features and high-level semantic concepts.

4. To reduce the semantic gap, concept-based multimedia search has been introduced. Select a concept lexicon relatively easy for computers to understand Collect training data Model high-level semantic concepts Introduction

5. Problem • Concept Lexicon selection is usually simplified by manual selection or totally ignored in previous works. • i.e. Caltech 101, Caltech 256, PASCAL --Implicitly favored those relatively “easy” concepts • MediaMill challenge concept data (101 terms) and Large-Scale Concept Ontology for Multimedia (LSCOM) (1,000 concepts) -- Manually annotated concept lexicon on broadcast news video from the TRECVID benchmark • All the lexica ignore the differences of semantic gaps among concepts. • No automatic selection is executed.

6. Concepts with smaller semantic gapsare likely to be better modeled and retrieved than concepts with larger ones. Very little research is found on quantitative analysisof semantic gap. What are the well-defined semantic concepts for learning? How to automatically find them? Problem

7. Objective • Automatically construct a lexicon of high-level concepts with small semantic gap (LCSS). • Two propertiesfor LCSS: • 1) Concepts are commonly used. --The words have high occurrence frequency within the descriptions of real-world images. • 2) Concepts are expected to be visually and semantically consistent. -- Images of these concepts have smaller semantic gaps (easy to be modeled for retrieval and annotation)

8. Idea • Web images have rich textual features --filename, title, alt text, and surrounding text. • Input titles and comments are good semantic descriptions of the images • These textual features are much closer to the semantics of the images than visual features.

9. Framework of LCSS Construction

10. Data Collection • 2.4 million web images from photo forums --Photo.net, Photosig.com etc • Photos have high quality and rich textual information • 64 dimensional global visual feature --color moments, color correlogram and color-texture moments

11. Framework of LCSS Construction

12. Nearest Neighbor Confidence Score (NNCS) • The higher the NNCS value, the smaller the semantic gap. • Calculate NNCS Score for all 2.4 million images with K=500 • Select 36231 candidate images with top NNCS --its relatively large size and memory concern for the Affinity Propagation clustering algorithm.

13. Framework of LCSS Construction

14. Clustering Using Affinity Propagation • Fast for large scale data set and require no prior information (e.g., number of clusters). • 36231×36231 content-context similarity matrix (CCSM) P

15. Framework of LCSS Construction

16. Text-based Keyword Extraction (TBKE) • The set of the related keywords of cluster is denoted as . • The relevance score of a keyword to cluster is denoted as: • For each keyword , its relevance score to the whole cluster pool C can be denoted as

17. Category Concepts Scene/Landscape sunset, sky, beach, garden, lake, sunflow, water firework, cloud, moon, sunrise, mountain, city, river, snow, rain, home, island Object flower, rose, butterfly, tree, bee, candle, bridge, leaf, eye, tulip, orchid, house, peacock, window, glass, bird, rock Color blue, red, yellow, green, pink, purple, orange, dark, golden Season fall, spring, summer, autumn Others small, wild LCSS Table 1. Top 50 keywords in the LCSS lexicon

18. Experiments • For each keyword w, randomly selected 500 titled photos with this keyword. • The confidence value decreases similarly to the keyword rank’s depreciation. • Demonstrates that the image labeled with the top words have higher confidence value. Figure 1: Distribution of average confidence value. x-axis represents top 15 keywords (from left to right) in LCSS.

19. Experiments • Apply the lexicon on the University of Washington (UW) dataset (1109 images, 5 labeled ground truth annotations, 350 unique words) • Refine the annotation results obtained by the search-based image annotation algorithm(SBIA). Figure 2: Image annotation refinement scenario .

20. Experiments Figure 3: Annotation precision and recall of different sizes of lexicon . • The refinement distinctively improves original annotation’s precision and recall when s becomes larger. • The performance of refinement keeps stable when s is equal or larger than 100.

21. Experiments Figure 4: Annotation precision and recall of different sizes of m . • When m is ranging from 3 to 7, the Precision and Recall of refined annotation are improved most . • The Precision and Recall of the annotation pruning remains same especially while m is larger than 7. --most of top 7 annotation words fall into the LCSS.

22. Experiments Figure 4: Annotation precision and recall of different lexica. • LSCOM: Large Scale Concepts Ontology for Multimedia. WordNet: A very large lexical database of English. (100,303) • LSCOM and WordNet do not improve the annotation precision. --Many correct annotations are not included in these two lexica thus are pruned.

23. Conclusion • Quantitatively study and formulate the semantic gap problem. • Propose a novel framework to automatically selectvisually and semantically consistent concepts. • LCSS is the first lexicon for concepts with small semantic gap • great help for data collectionandconcept modeling. • a candidate pool of semantic concepts --image annotation, annotation refinementandrejection. • Potential applicationsin query optimization and MIR.

24. Future Work • Investigate different features to construct feature-based lexica • Texture feature • Shape feature • SIFT feature • Open questions • How many semantic concepts are necessary? • Which features are good for image retrieval with specific concept? • ……

25. Reference • C. G. Snoek, M. Worring, J. C. van Gemert, J. M. Geusebroek, and A. W. Smeulders. “The challenge problem for automated detection of 101 semantic concepts in multimedia.” Proc. of ACM Multimedia, 2006. • C. M. Naphade, J. R. Smith, J. Tesic, and S. F. Chang, et al. “Large-scale concept ontology for multimedia.” IEEE MultiMedia, 13(3):86–91, 2006. • B. J. Frey, and D. Dueck. Clustering by passing messages between data points. Science, 315:972-976, 2007. • X. J. Wang, L. Zhang, F. Jing, and W. Y. Ma. “AnnoSearch: image auto-annotation by search”. Proc. of IEEE Conf. CVPR, New York, June, 2006. • C. Wang, F. Jing, L. Zhang, and H. J. Zhang. “ Scalable search-based image annotation of personal images. ” Proc. of the 8th ACM international Workshop on Multimedia information Retrieval, Santa Barbara, CA, USA, 2006. 10.