1 / 13

MUMIS

MUMIS. Multimedia Indexing and Searching. Franciska de Jong & Thijs Westerveld University of Twente westerve@cs.utwente.nl. OBJECTIVES. Automatically indexing of video Data from different media sources (paper, radio, tv) Domain: soccer Digitise + ASR Extract significant events

kato
Download Presentation

MUMIS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MUMIS Multimedia Indexing andSearching Franciska de Jong & Thijs Westerveld University of Twente westerve@cs.utwente.nl

  2. OBJECTIVES • Automatically indexing of video • Data from different media sources (paper, radio, tv) • Domain: soccer • Digitise + ASR • Extract significant events • Merge annotations • Store final annotations • UI for searching

  3. FACTS SHEET Title: MUMIS: Multimedia Indexing and Searching Environment Funding: EU Language Engineering Sector of TAP Duration: 30 months July 2000 – January 2003 Volume: 2.4 M Euro, 385 Person months Languages:Dutch, English, German (Swedish)

  4. Consortium • University of Twente (NL) • Sheffield University (UK) • University of Nijmegen (NL) • DFKI LT-Lab (DE) • Max Planck Institute for Psycholinguistics (DE) • Esteam (SE) • VDA(NL)

  5. Information Extraction Merging DE NL EN IE Merged Annotated formal text ASR Merging Annotations Automatic Speech Recognition Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Formal Text Free Text Speech Transcr Formal Text Formal Text Formal Text Formal Text Speech Signals Formal Text Formal Text Formal Text Formal Text Formal Text Anno-tations Offline Processing

  6. ENTITY RELATION EVENT Time Date Person Score Object Foul Goal Player:… Cause:… Time:… ... Artifact Player Official Defender Stopper DOMAIN MODELLING DATA: text, video, audio Annotations Multilingual IE Multilingual Search ... Location Player:… Consequence:… Time:… Location:... <?xml version=…> <mumis-ontology> <version>…</version> ... <class> <name>Defender</name> <documentation>a ’Defender’ is a …</documentation> <subclass-of>Player</subclass-of> </class> </mumis-ontology> Multilingual Lexicons

  7. SPEECH RECOGNITION • Large-vocabulary • Speaker independent • Phoneme-based • Hidden Markov models • acoustic model • language model • Emotionally coloured speech • Domain language model • Match specific vocabularies (player names)

  8. INFORMATION EXTRACTION • multilingual • formal descriptions • closed captions • tickers • newspapers • ASR output (radio/TV comment)

  9. IE DATA Ticker 24 Scholes beats Jens Jeremies wonderfully, dragging the ball around and past the Bayern Munich man. He then finds Michael Owen on the right wing, but Owen's cross is poor. Formal text Schoten op doel 4 4 Schoten naast doel 6 7 Overtredingen 23 15 Gele kaarten 1 1 Rode kaarten 0 1 Hoekschoppen 3 5 Buitenspel 4 1 TV report Scholes Past Jeremies Owen Newspaper Owen header pushed onto the postDeisler brought the German supporters to their feet with a buccaneering run down the right. Moments later Dietmar Hamann managed the first shot on target but it was straight at David Seaman. Mehmet Scholl should have done better after getting goalside of Phil Neville inside the area from Jens Jeremies’ astute pass but he scuffed his shot.

  10. IE Techniques & resources 24 Scholes beats Jens Jeremies wonderfully, dragging the ball around and past the Bayern Munich man. He then finds Michael Owen on the right wing, but Owen's cross is poor. • Tokenisation • Lemmatisation • POS + morphology • Named Entities • Shallow parsing • Co-reference resolution • Template filling He then finds Michael Owen on the right wing PASS player1 = Scholes player2 = Owen. He then finds VP Michael Owen on the right wing NP but Owen's cross NP 24 Scholes beats Jens Jeremies wonderfully , dragging ... 24 Scholes beat Jens Jeremies wonderfull , drag ... 24 NUM Scholes PROP beat VERB 3p sing Jens PROP Jeremies PROP wonderfull ADV , PUNCT ... He Scholes then finds Michael Owen on the right wing … 24 time Scholes player beat Jens Jeremies player wonderfull , …

  11. MERGING • Fuse annotations and recover from errors and differences: • Multiple annotations of the same event (possibly with different attributes, e.g. time). • Wrong event descriptions because of information extraction errors. • Merging multiple partial annotations, e.g. by solving unsolved references like “star player”. • Description logic

  12. Search for interesting events with formal questions (user interface in many languages) Indicate hits by thumbnails & let user select scene Play scene via the Internet & allow scrolling PSV - Ajax 1995 Ned - Eng 1998 Ned - Ger 1998 ON-LINE TASKS Multilingual Search and Display Give me all goals from Overmars shot with his head in 1. Half. Event=Goal; Scorer=Overmars; Cause=Head; Time<=45

  13. SUMMARY • Multimedia and multilingual • ASR on emotionally coloured speech • IE on ASR output • Merging different annotations • Search archives and play video online http://parlevink.cs.utwente.nl/projects/mumis.html

More Related