Bridge Semantic Gap: A Large Scale Concept Ontology for Multimedia (LSCOM)

1. Bridge Semantic Gap: A Large Scale Concept Ontology for Multimedia (LSCOM) Guo-Jun Qi Beckman Institute University of Illinois at Urbana-Champaign

2. LSCOM (Large Scale Concept Ontology for Multimedia) • A broadcast news video dataset • 200+ news videos/ 170 hours • 61,901 shots • Language • English/Arabic/Chinese

3. Why broadcast News ontology? • Critical mass of users, content providers, applications • Good content availability (TRECVID LDC FBIS) • Share Large set of core concepts with other domains

4. LSCOM Provides • Richly annotated video content for accomplishing required access and analysis functions over massive amount of video content • Large scale useful well-defined semantic lexicon • More than 3000 concepts • 374 annotated concepts • Bridging semantic gap from low-level features to high-level concepts

5. A LSCOM concept 000 - Parade Concept ID: 000Name: ParadeDefinition: Multiple units of marchers, devices, bands, banners or Music.Labeled: Yes

6. LSCOM Hierarchy • http://www.lscom.org/ontology/index.html Thing .Individual ..Dangerous_Thing ...Dangerous_Situation ....Emergency_Incident .....Disaster_Event ......Natural_Disaster ....Natural_Hazard .....Avalance .....Earthquake .....Mudslide .....Natural_Disaster .....Tornado ...Dangerous_Tangible_Thing ....Cutting_Device

7. Definition: What’s the ontology? (Wikipedia) • An ontology is a formal representation of the knowledge by a set of concepts within a domain and the relationships between those concepts. It is used to reason about the properties of that domain, and may be used to describe the domain.

8. Ontology • Represents the visual knowledge base in a structure way • Graph structure • Tree (hierarchy) structure • Images/videos can be effectively learned and retrieved by the coherence between concepts • Logical coherence • Statistical coherence

9. An Ontology Hierarchy: Military Vehicle

10. An example from Wikipedia

11. Ontology Tree for LSCOM

12. A Light Scale Concept Ontology for Multimedia Understanding (LSCOM-Lite) • The aim is to break the semantic space using a few concepts (39 concepts). • Selection Criteria • Semantic Coverage • As many as semantic concepts in News videos could be covered by the light concept set. • Compactness • These concept should not semantically overlap. • Modelability • These concepts could be modeled with a smaller semantic gap.

13. Selected concept dimensions • Divide the semantic space into a multimedia-dimensional space, where each dimension is nearly orthogonal • Program Category • Setting/Scene/Site • People • Objects • Activities • Events • Graphics

14. Histogram of LSCOM-Lite Concepts

15. Some example keyframes

16. Applications • Application I: Conceptual Fusion (most basic – early fusion) • Application II: Cross-Category Classification (inter-class relation) • Application III: Event Dynamic in Concept Space

17. Application I: Conceptual Fusion Concept 1 Concept 2 Video Concept 3 Classifier … Concept n Visual Features

18. LSCOM 374 Models • 374 LIBSVM models • http://www.ee.columbia.edu/ln/dvmm/columbia374/ • Feature used (MPEG-7 descriptors) • Color Moments • Edge Histogram • Wavelet Texture • LIBSVM – a library for support vector machine at http://www.csie.ntu.edu.tw/~cjlin/libsvm/

19. Application II: cross-category classification with concept transfer G.-J. Qi et al. Towards Cross-Category Knowledge Propagation for Learning Visual Concepts, in CVPR 2011

20. Instance-Level Concept Correlation Mountain Castle +1 +1 Mountain and castle Castle only Mountain only -1 -1

21. Transfer Function Mountain, Castle Mountain Castle None of them

22. Model Concept Relations

23. Automatically construct ontology in a data-driven manner

24. An application III – Event Dynamics in Concept Space

25. Event Detection with Concept Dynamics W. Jiang et al, Semantic event detection based on visual concept prediction, ICME, Germany, 2008.

26. Open Problems • Cross-Dataset Gap • Generalize LSCOM dataset to other dataset (e.g., non-news video dataset) • Cross-Domain Gap • Text script associated with news videos • Can help information extraction for visual concepts? • Automatic ontology construction • Task dependent v.s. task independent • Data driven v.s. preliminary knowledge (e.g., WordNet) • Incorporate prior human knowledge (logic relation etc.)

27. TRECVID Competition • Task 1: High-Level Feature Extraction • Input: subshot • Output: detection results for 39 LSCOM-Lite concepts in the subshot

28. High-Level Feature Extraction • Each concept assumed to be binary (absent or present) in each subshot • Submission: Find subshots that contain a certain concept, rank them by the detection confidence score, and submit the top 2000. • Evaluations: NIST evaluated 20 medium frequent concepts from 39 concepts using a 50% random samples of all the submission pools

29. 20 Evaluated Concepts

30. Evaluation Metric: Average Precision • Relevant subshots should be ranked higher than the irrelevant ones. R is the number of relevant images in total, Rj is the number of relevant images in top j images, Ij indicates if the jth image is irrelevant or not.

31. Results

32. TRECVID Competition • Task II: Video Search • Input: text-based 24 topics • Output: relevant subshots in the database

33. Topics to search

34. Topics to search (cont’d)

35. Topics to search

36. Three Types of Search Systems

37. Results: Automatic Runs

38. Results: Manual Runs

39. Results: Interactive Runs

40. Machine Problem 7: Shot Boundary Detection in Videos

41. Goals • Detect the abrupt content changes between consecutive frames. • Scene changes • Scene cuts

42. Steps • Step 1: Measuring the change of content between video frames • Visual/Acoustic measurements • Step 2: Compare the content distance between successive frames. If the distance is larger than a certain threshold, then a shot boundary may exist.

43. Measuring Content based on Visual Information • 256 dimensional Color Histogram • In RGB space, normalize the r, g, b in [0,1] • Color space nr 8X8 histogram ng

44. Color Histograms • Divide each image into four parts, each part has a 8X8 histogram, and 256 dim features in total.

45. Acoustic Features • 12 cepstral coefficients • Energy (sum of square of raw signals) • Zero crossing rates (ZCR) ZCR = sum(|sign(S(2:N))-sign(S(1:N-1))|) • Hints: normalize energy to avoid it over-dominating when computing distances between successive frames

46. Datasets • Two videos of little over one minute • Manually label the shot boundary

47. What to submit • Source code • Report • compare shot boundary detection results returned by your algorithm with the manually labeled boundaries • Compare • Explain your choice of threshold • Explain the differences between the acoustic-based and visual-based detection results

48. Where and when to submit • Email to ece.ece.ece.417@gmail.com • Due: May 2nd

49. Thanks! Q&A