1 / 24

Outline

LREC 2010 Speech Grammars for Textual Entailment Patterns in Multimodal Question Answering Daniel Sonntag , Bogdan Sacaleanu, DFKI 21/05/2010. Outline. Semantic Dialogue Shell Textual Entailment Processing Example Conclusions. | 2. Acknowledgements.

austin
Download Presentation

Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LREC 2010Speech Grammars for Textual Entailment Patterns in Multimodal QuestionAnsweringDaniel Sonntag, Bogdan Sacaleanu, DFKI21/05/2010 Daniel Sonntag

  2. Outline Semantic Dialogue Shell Textual Entailment Processing Example Conclusions | 2 Daniel Sonntag

  3. Acknowledgements • Thanks go out to Robert Nesselrath, Yajing Zang, Günter Neumann, Matthieu Deru, Simon Bergweiler, Gerhard Sonnenberg, Norbert Reithinger, Gerd Herzog, Alassane Ndiaye, Tilman Becker, Norbert Pfleger, Alexander Pfalzgraf, Jan Schehl, Jochen Steigner, and Colette Weihrauch for the implementation and evaluation of the dialogue infrastructure. SPARQL Ease the interface to external third-party components. Robust Question Understanding Daniel Sonntag

  4. Semantic Dialogue Shell | 4 Daniel Sonntag

  5. Dialogue Shell Workflow Speech Interpretation Modality Fusion Text Interpretation Text Summarisation Interactive Semantic Mediator Gesture Interpretation External Information Sources Remote Linked Data Services • - Domain Model • Context Model • User Model SPARQL Dialogue and Interaction Management eTFS/SPARQL Semantiic (Meta) Services Graphic Generation SPARQL RDF KOIOS (Yago Ontology) Personalisation Interactive Service Compo-sition Text Generation SPARQL OWL AOIDE (Music Ontology) OWL-API Speech Interpretation Presentation Planning Visualisation Visualisation Service Daniel Sonntag

  6. THESEUS’s Semantic Dialogue Shell: Goals and Requirements • Multimodal interaction with the Semantic Web and the Internet of Services • Components customisable to different use case scenarios • Flexible adaptation to • Input and output modalities • Interaction devices • Knowledge bases • To understand a greater number of queries: • Robust question understanding (NLU) when using both speech and written text input • Semantic (i.e., a RDF or OWL based) query interpretation • The combination of robust question understanding and ontology-based answer retrieval Daniel Sonntag

  7. SPARQL Query Editor SPARQL Daniel Sonntag

  8. Daniel Sonntag

  9. Speech Grammar <utterance name="SHOW_CV_OF_PERSON"> <phrases> <phrase>zeige ?mir den [werdegang lebenslauf] [von zu] PERSON</phrase> <phrase>sage ?mir mehr über den [werdegang lebenslauf] von PERSON</phrase> <phrase>wie ist der [werdegang lebenslauf] von PERSON</phrase> </phrases> <semantic-interpretation> <object type="odp#TaskRequest"> <slot name="odp#fusion-confidence"> <value type="Float">1.0</value> </slot> <slot name="odp#hasContent"> <object type="dialogshell#BackendRetrievalTask"> <slot name="dialogmanager#backendComponent"> <value type="String">mediator:summarizer</value> </slot> <slot name="odp#hasContent"> <variable name="PERSON"/> </slot> </object> </slot> </object> </semantic-interpretation> </utterance> Daniel Sonntag

  10. Textual Entailment Our idea is that an NLU grammar for speech input can be reused to build more robust multimodal text-based question understanding by automatically generating textual entailment patterns. | 10 Daniel Sonntag

  11. Textual Entailment & Information Access Conceptual Method2 Implicit Mapping Reasoning Ontology (RDF/OWL) Method1 ? • Method1: Speech / Semantic Grammars • RDF/OWL reasoning • Method2: RTE • textual reasoning Request Information (RDF) Textual Daniel Sonntag

  12. Textual Entailment through Alignments • For textual entailment to hold we need: • text AND background knowledgehypothesis • but backgroundknowledge should not entail hypothesis alone • Background Knowledge • Unsupervised acquisition of linguistic and world knowledge from general corpora and web • Acquiring larger entailment corpora • Manual resources and knowledge engineering • Alignment-based TE and Background Knowledge • Preprocessing: POS, morphology, cognates • Representation: bag-of-words • Knowledge Sources: WordNet, Roget‘s Thesaurus, Wehrle Thesaurus Daniel Sonntag

  13. Argumentation • Input modalities are usually interpreted according to separate models and aligned to a shared model (often coarse-grained). • Present a method of interpretation based on a common model (propagated changes to multiple modalities). • Built on the grammar for speech inputs = Leveraging Existing Speech Grammar Knowledge Daniel Sonntag

  14. Processing Example | 14 Daniel Sonntag

  15. Entailment Patterns and Possible Hypotheses Daniel Sonntag

  16. Association-based word alignment. Three steps: lexical segmentation, when boundaries of lexical items are identified; correspondence, when possible similarities are suggested in line with some correspondence measures; alignment, when the most likely semantically similar word is chosen. Entailment Patterns and Alignment Engine Daniel Sonntag

  17. Question: What is the birthplace of Angela Merkel? Pattern: Where is Angela Merckel born? Filters on a full alignment. Entailment Patterns and Alignment Techniques Daniel Sonntag

  18. Entailment Patterns and Alignment Techniques POS Filter: Exclude unlikely alignments based on POS. Allow for the additional mappings: verb to noun (i.e., born vs. birthplace) Lexical Semantic Resource Filter: WordNet (synonyms); Roget Thesaurus (conceptually related words) String Similarity Filter: Dice coefficient, Longest common subsequence ratio; submatches, misspellings System of weights: nouns, verbs, and adjectives are better scored than function words. Daniel Sonntag

  19. Dialogue Example • (1) U: “Open my personal address book. What do you know about Claudia?” • (2) S: “There’s an entry: Claudia Schwartz. The personal details are shown below. She lives in Berlin.” + Google Map Display of street coordinates. • (3) U: “Which is Claudia’s favorite kind of music? Do you know the bands she likes most?” • (4) S: “Nelly Furtado” + Displays videos obtained from YouTube. (Rest API) • (5) U: “How did experts rate her last album?” • (6) S: Shows an expert review according to the BBC Linked Data Set. • (7) U: “Show me other news.” • (8) S: Opens a browser + Text field and a new agency Internet page (featuring Angela Merkel) • (9) U writes: “Where was Angela Merkel born? / In which town was Angela Merkel born?” etc. • (10) S: “She was born in Hamburg.” • (11) U speaks again: “And Barack Obama?” • (12) S: “He was born in Honolulu.” • (13) U: “Show me Angela Merkel’s career.” Daniel Sonntag

  20. Touchscreen Installation Daniel Sonntag

  21. Image Analysis in Biomedicine MEDICO Retrieval and examination of 2D picture series Daniel Sonntag

  22. Conclusions | 22 Daniel Sonntag

  23. Conclusions We described a multimodal dialogue shell for QA and focussed on the robust multimodal question understanding task. The textual interpretation is based on automatically generated textual entailment patterns. As a result, we can deal with written text input and different surface forms more flexibly according to the derived entailment patterns. | 23 Daniel Sonntag

  24. Method Comparison 2 1 Daniel Sonntag • Method 1: Speech Grammars • Speech grammars are verbose • Requires full coverage of expected input • Hard-coded reasoning in rules • Example: • Show me all pictures of X. • What pictures does X have? • Show me all images of X. • Method 2: NLU Grammars • Use of Textual Entailment • NLU grammars are compact • Requires partial coverage of possible input • Example: • Show me all pictures of X. • Entailed utterances: • What pictures does X have? • Show me all images of X.

More Related