1 / 20

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment. Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield

kane-hebert
Download Presentation

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment Thierry Declerck, Peter Wittenburg and Hamish Cunningham DFKI GmbH, Max-Planck-Institut für Psycholinguistik and University of Sheffield ACL/EACL2001 Workshop on Human Language Technology and Knowledge Management

  2. The MUMIS Consortium • CTIT University of Twente, Enschede, NL NLP/IE • TSI University of Nijmegen, Nijmegen, NL ASR • DFKI Saarbrücken, D NLP/IE • MPI Nijmegen, NL Online SW • DCS University of Sheffield, UK NLP/IE • ESTEAM Gothenburg, SE (location Athens, GR) Translation Software • VDA Hilversum, NL Dissemination

  3. Objectives • Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process) Find and annotate relevant events, together with the involved entities and relations • Technology development to exploit indexed multimedia archives (on-line process) Search for interesting scenes and play them via Internet • Test Domain: Soccer Games / UEFA Tournament 2000

  4. Off-line Task • Automatic Speech Recognition (Radio/TV Broadcasts) Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German) • Natural Language Processing (Information Extraction) Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events • Merging all the annotations produced so far • Create a database with formal annotations

  5. The Generation of Formal Annotations Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences All Formal Annotations available in XMLStandard

  6. The Event Table Related to domain ontology and multilingual terminology. Guiding the generation of formal annotations

  7. Close caption 3 Languages multilingual IE => event tables Event = goal Type = Freekick Player = Basler Dist. = 25 m Time = 17 Score: leading Event = goal Player= Basler Team = Germany Time = 18 Score = 1:0 Finalscore = 1:0 Merging of Annotations Event = goal Player = Basler Dist. = 25 m Time = 18 Score = 1:0 Newspaper Text Newspaper Text Newspaper Text Newspaper Text Newspaper Text Newspaper Text Newspaper Texts 3 Languages Radio Commenting 3 Languages • Freekick • Goal • Pass • Defense Radio Commenting 3 Languages Radio Commenting 3 Languages Audio Commenting (TV, Radio) 3 Languages • 17 min • 18 min • 24 min • 28min • 1:0 Events indexed in video recording • Foul • Freekick • Dribbling • Neville • Basler • Matthäus • Campbell • Basler • Scholl • 25 m • 25 m • 60 m Off-line Task Event = goal Type = Freekick Player = Basler Team = Germany Time = 18 Score = 1:0 Final score = 1:0 Distance = 25 m

  8. The Role of IE in MUMIS • Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user. • The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS) • IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).

  9. Extension of our IE system in MUMIS • Multilingual and multisource IE. Incremental information building • Cross-document co-reference resolution • Combine Metadata and event extraction => better organisation and dynamic updating of information (KM) • Multiple presentation of results: Template, Event table and Hyperlinks (Named Entities, rel. to KM)

  10. Example of Processing Formal Texts • Formal Text • The Formal Text annotated with domain-specific information

  11. Example of Processing Semi-Formal Texts • Semi-Formal Text • The Semi-Formal Text annotated with domain-specific information

  12. Merging Component • Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those • Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology • Completing the information with domain knowledge, inference Machine

  13. On-line Tasks • Searching and Displaying • Search for interesting events with formal queries Give me all goals from Overmars shot with his head in 1. Half. Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball • Indicate hits by thumbnails & let user select scene • Play scene via the Internet & allow scrolling etc Of course: slow motion, fast play, start/stop, etc • User Guidance (Lexica and Ontology)

  14. On-line Tasks • Freekick • Goal • Pass • Defense Knowledge Guided User Interface & Search Engine • 17 min • 18 min • 24 min • 28min • 1:0 Play Movie Fragment of that Game • Foul • Freekick • Dribbling • Neville • Basler • Matthäus • Campbell • Basler • Scholl • 25 m • 25 m • 60 m München - Ajax 1998 München - Porto 1996 Deutschland - Brasilien 1998 Prototype Demo

  15. More about MPEG (Moving Picture Coding Experts Group) • MPEG-1: low-level media encoding and compression format (VHS quality, ~ 2-3 Mbps - good for streaming) • MPEG-2: improved media encoding and compression format (S-VHS quality, ~ 5-10 Mbps, digital TV and DVD standard • MPEG-4: Codes content as objects and enables those objects to be manipulated at the client, optimized compression

  16. Ontology Client Objects Lexica Client Applet JMF Media Server Objects Query Engine Objects HTTP RMI RMI (RTP, RTSP) WWW Server Java Server MPEG Movies Keyframes Annotations Metadata File Server Media Server MPEG1 DB Server rDBMS Media Server MPEG1 Media Server MPEG1 Media Server MPEG1 JDBC On-line SW Architecture • Client-Server structure: • fully distributed • JMF media presentation • RMI-based interaction • Query interface: • automatic pre-selection • guided by domain knowledge • interactive, visual feedback

  17. Media Server RAID 1Gbps Gb-Switch Router FC Switch GB Switch Tape Library Internet Media Server On-line HW Architecture • efficient & reliable storage management • (near-line capacity, media change, 2. Location) • high storage capacity (n TB, 1 h MPEG1 = 1 GB) • powerful media servers / powerful network

  18. UI / Annotation • UI optimization • thumbnails not that informative • which thumbnail? (several around time mark) • automatic thumbnail adjustment? • seamless operation for user • lexicon/ontology guidance • user driven input • Manual annotation tools • MediaTagger • EUDICO

  19. Gain - User Group • What gets lost? Is it necessary? • Potential: direct Internet Service, less dependencies

  20. Acknowledgements • UEFA • DFB, FA, KNVB • EBU, WDR, NOS, SWR Allez les Bleus!!

More Related