1 / 28

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment. Thierry Declerck DFKI GmbH Annotation Workshop, DI, 15. Februar 2002. The MUMIS Consortium. CTIT University of Twente, Enschede, NL NLP/IE TSI University of Nijmegen, Nijmegen, NL ASR

charis
Download Presentation

The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Automatic Generation of Formal Annotations in a MultiMedia Indexing and Searching Environment Thierry Declerck DFKI GmbH Annotation Workshop, DI, 15. Februar 2002

  2. The MUMIS Consortium • CTIT University of Twente, Enschede, NL NLP/IE • TSI University of Nijmegen, Nijmegen, NL ASR • DFKI Saarbrücken, D NLP/IE • MPI Nijmegen, NL MM Archives • DCS University of Sheffield, UK NLP/IE • ESTEAM Gothenburg, SE (location Athens, GR) Translation Software • VDA Hilversum, NL Video

  3. Objectives of MUMIS • Technology development to automatically index (with formal annotations) lengthy multimedia recordings (off-line process) Find and annotate relevant events, together with the involved entities and relations. Also detect Metadata information. • Technology development to exploit indexed multimedia archives (on-line process) Search for interesting scenes and play them via Internet • Test Domain: Soccer Games / UEFA Tournament 2000

  4. Off-line Task • Indexing by • Automatic Speech Recognition (Radio/TV Broadcasts) Automatically transforms the speech signals into texts (for 3 languages — Dutch, English and German) • Natural Language Processing (Information Extraction) Analyse all available textual documents (newspapers, speech transcripts, tickers, formal texts ...), identify and extract interesting entities, relations and events. Also detect Metadata information. • Merging all the annotations produced so far • Create a database with formal annotations • Use video processing to adjust time marks

  5. Gain • What gets lost? Is it necessary? • Potential: direct Internet Service, less dependencies

  6. The Generation of Formal Annotations Metadata (type of game, teams, date, final score, players etc.), as they can be used a.o. for classifying and filtering videos in the MM digital archive Events (particular actions with time codes, involved entities and related events), as they can be extracted from the video sequences All Formal Annotations available in XMLStandard

  7. The Event Table Related to domain ontology and multilingual terminology. Guiding the generation of formal annotations

  8. Tickers etc. 3 Languages multilingual IE => event tables Event = goal Type = Freekick Player = Basler Dist. = 25 m Time = 17 Score: leading Event = goal Player= Basler Team = Germany Time = 18 Score = 1:0 Finalscore = 1:0 Merging of Annotations Event = goal Player = Basler Dist. = 25 m Time = 18 Score = 1:0 Newspaper Text Newspaper Text Newspaper Text Newspaper Text Newspaper Text Newspaper Text Newspaper Texts 3 Languages Radio Commenting 3 Languages • Foul • Goal • Pass • Defense Radio Commenting 3 Languages Radio Commenting 3 Languages Audio Commenting (TV, Radio) 3 Languages • 17 min • 18 min • 24 min • 28min • 1:0 Events indexed in video recording • Freekick • Dribbling • Neville • Basler • Matthäus • Campbell • Basler • Scholl • 25 m • 25 m • 60 m Off-line Task Event = goal Type = Freekick Player = Basler Team = Germany Time = 18 Score = 1:0 Final score = 1:0 Distance = 25 m

  9. The Role of IE in MUMIS • Information Extraction (IE) is the task of identifying, collecting and normalizing relevant information for a specific application or user. • The relevant information is typically represented in form of predefined “templates”, which are filled by means of Natural Language (NL) analysis (Template = Event Table in MUMIS) • IE combines pattern matching mechanisms, (shallow) NLP and domain knowledge (terminology and ontology).

  10. Extension of our IE system in MUMIS • Multilingual and multisource IE. Incremental information building • Cross-document co-reference resolution • Combine Metadata and event extraction => better organisation and dynamic updating of information (KM) • Multiple presentation of results: Template, Event table, integration in MPEG-7 XML and Hyperlinks (Named Entities, rel. to Knowledge Management)

  11. The DFKI Implementation • Based on XML output of SPPC (Dev. At DFKI) • Mapping the XML into a feature structure (the CorpA/schug Program) • Cascaded grammar descriptions for enriching (or correcting) the SPPC output • Including agreement processing and detection of grammatical functions • Adapting the “Paradime triangle” for template generation and filling

  12. Information Extraction • IE is generally subdivided in following tasks: - Named Entity task (NE) - Template Element task (TE) - Template Relation task (TR) - Scenario Template task (ST) - Co-reference task (CO)

  13. Subtasks of IE • Named Entity task (NE): Mark into the text each string that represents, a person, organization, or location name, or a date or time, or a currency or percentage figure. • Template Element task (TE): Extract basic information related to organization, person, and artifact entities, drawing evidence from everywhere in the text.

  14. Subtasks of IE (2) • Template Relation task (TR): Extract relational information on employee_of, manufacture_of, location_of relations etc. (TR expresses domain-independent relationships). • Scenario Template task (ST): Extract pre-specified event information and relate the event information to particular organization, person, or artifact entities (ST identifies domain and task specific entities and relations). • Co-reference task (CO): Capture information on co-referring expressions, i.e. all mentions of a given entity, including those marked in NE and TE.

  15. IE applied to soccer Terms as descriptors for the NE task Team:TitelverteidigerBrasilien, den respektlosen AußenseiterSchottland Player:SuperstarRonaldo, von BewacherCalderwood noch von AbwehrchefHendry, von Jacksonals drittem Stürmer,Torschütze Cesar, von Roberto Carlos(16.), Referee: vom spanischen SchiedsrichterGarcia Aranda Trainer: Schottlands Trainer Brown, Kapitän Hendry seinen KeeperLeighton Location: im Stade de France vonSt. Denis (more fine-grained location detection would be: Stadion: im Stade de FranceandCity: vonSt. Denis ) Attendance: Vor 80000Zuschauern

  16. IE applied to soccer (2) Terms for NE Task Time: in der 73. Minute, nach gerade einmal 3:50 Minuten, von Roberto Carlos (16.),nach einer knappen halben Stunde, scheiterte Rivaldo (49./52.) jeweils nur knapp, das vor der Pause Versäumte versuchten die Brasilianer nach Wiederbeginn, ... Date: am Mittwoch, der Turnierstart (?), im WM-Eröffnungsspiel(?) Score/Result: Brasilien besiegt Schottland 2:1, einen 2:1 (1:1)-Sieg, der zwischenzeitliche Ausgleich, in der 4. Minute in Führung gebracht, köpfte zum 1:0 ein

  17. IE applied to soccer (3) Relations for TR Task Opponents: BrasilienbesiegtSchottland, feierteder Top-Favorit ... einen glücklichen 2:1 (1:1)-Siegüber den respektlosen Außenseiter Schottland, Player_of: hatte Cesar Sampaioden vierfachen Weltmeister ... in Führung gebracht, Collins gelang ... der zwischenzeitliche Ausgleichfür die Schotten, der KeeperdesFC Aberdeen, BrasiliensKeeper Taffarel Trainer_of: Schottlands Trainer Brown ...

  18. IE applied to soccer (4) Events for ST task: Goal: in der 4. Minute in Führung gebracht, das schnellste Tor ... markiert, Cesar Sampaio köpfte zum 1:0 ein, Collins (38.) verwandelte den Strafstoß, hätte Kapitän Hendry seinen Keeper Leighton um ein Haar zum zweiten Mal bezwungen, von dem der Ball ins Tor prallte Foul: als er den durchlaufenden Gallacher im Strafraum allzu energisch am Trikot zog Substitution: und mußte in der 59. Minute für Crespo Platz machen...

  19. team-template TACTIC [ ] SCORE [ ] NAME [ ] PLAYER [ ] TRAINER [ ] IE applied to soccer (5) Description of the Templates: Team team-template TACTIC [ ] SCORE [S] NAME [ ] PLAYER [P] TRAINER [ ] goal-template TIME [ ] SCORE [S] PLAYER [P] TEAM [team-templ ] TYPE [ ] SUCCESS [ ]

  20. Merging Component • Acting on the generated formal annotations (Metadata and Events), but also interleaving with the generation process of those • Checking consistency, eliminating redundancy (Template Merging), in accordance with domain ontology • Completing the information with domain knowledge, inference Machine

  21. Use of Standards • XML as the annotation language and data interchange format • MPEG-7: standard for the description of features of multimedia content, XML compliant (for content description), with a slot for textual annotations

  22. More about MPEG (Moving Picture Coding Experts Group) • MPEG-1: For the storage and retrieval of movie pictures and audio on storage media • MPEG-2: For digital television • MPEG-4: Codes content as objects and enables those objects to be manipulated • MPEG-7: Where 1,2 and 4 make content available, MPEG-7 allows to find the content one needs

  23. On-line Tasks • Searching and Displaying • Search for interesting events with formal queries Give me all goals from Overmars shot with his head in 1. Half. Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball • Indicate hits by thumbnails & let user select scene • Play scene via the Internet & allow scrolling Of course: slow motion, fast play, start/stop, etc

  24. On-line Tasks • Searching and Displaying • Search for interesting events with formal queries Give me all goals from Overmars shot with his head in 1. Half. Event=Goal; Player=Overmars; Time<=45; Previous-Event=Headball • Indicate hits by thumbnails & let user select scene • Play scene via the Internet & allow scrolling Of course: slow motion, fast play, start/stop, etc

  25. On-line Tasks Knowledge Guided User Interface & Search Engine München - Ajax 1998 München - Porto 1996 Deutschland - Brasilien 1998 Play Movie Fragment of that Game

  26. Ontology Client Objects Lexica Client Applet JMF Media Server Objects Hit Rendering Objects Query Engine Objects HTTP RMI RMI (RTP, RTSP) WWW Server Java Server MPEG Movies Keyframes Media Server MPEG1 File Server DB Server rDBMS Media Server MPEG1 Media Server MPEG1 Media Server MPEG1 JDBC Annotations Metadata On-line SW Architecture • Server structure: • fully distributed • JMF media presentation • RMI-based interaction • Query interface: • pre-selection • guided by domain knowledge • interactive, visual feedback

  27. Media Server RAID 1Gbps Gb-Switch Router FC Switch GB Switch Tape Library Internet Media Server On-line HW Architecture • efficient & reliable storage management • (near-line capacity, media change, 2. Location) • high storage capacity (n TB, 1 h MPEG1 = 1 GB) • powerful media servers / powerful network

  28. Acknowledgements • UEFA • DFB, FA, KNVB • EBU, WDR, NOS, SWR

More Related