1 / 24

Multimodal research in Linköping

Multimodal research in Linköping. Arne Jönsson NLPLAB/HCS/IDA. NLPLAB. Upper row: Pernilla , Sonia, Lena, Annika, Micke, Magnus Bottom row: Håkan , Lars A, Robert, Mustapha, Arne , Lise, Anders, Lars D Missing: Nils , Bertil, Pontus, Gen. Current research.

kemp
Download Presentation

Multimodal research in Linköping

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal research in Linköping Arne Jönsson NLPLAB/HCS/IDA

  2. NLPLAB Upper row: Pernilla, Sonia, Lena, Annika, Micke, Magnus Bottom row: Håkan, Lars A, Robert, Mustapha, Arne, Lise, Anders, Lars D Missing: Nils, Bertil, Pontus, Gen

  3. Current research • (Multimodal) information access • Natural language interaction • Text and document analysis • Usability of multimodal interfaces • Open-Source and development methods

  4. Structured Information Dialogue system Dialogue system Interpreter Dialogue manager Generator

  5. Ex TV programme guide • Search for TV programs • Mediaterminal • Structured database • Spoken input • Developed from framework and method for iterative development • With Nokia Home Communication

  6. Demo

  7. Text and document analysis Unstructured documents Q&A Structured Information Language analysis MT

  8. Ex. AskRSV • Information extraction from text document • Simple Q&A-system • With Ida Infront

  9. 24 pages unstructured text Ex. taxation brochure

  10. THING EVENT ACTOR OBJECT PAPER DATE TAXATION TAX EVENT PEOPLE ORG. COMP. PUBL TAX-RETURN ASSET DEBT INCOME TRANSACTION Domain ontology in ASK-RSV • Taxonomy of a limited set of objects, events, properties and relations in the tax domain • Only IS-A and synonymy relations

  11. Unstructured documents Structured Information Dialogue system Language analysis Lexicon Grammar Ontology Combined system

  12. Shared knowledge sources • Ontology, lexicon and grammar • Different demands from IE and DS • Design • Development • Corpus analysis, user • Background system, database • Automatic update

  13. BirdQuest • Bird encyclopaedia • Tagged, templates • Corpus with user questions • Analysed • Based on dialogue systems framework

  14. Ötraf • Time tables from local bus company • Multimodal system • Dialogue • Handling of spatial and temporal expressions

  15. Studies of multimodal interaction • Wizard of Oz-simulation • Feedback study • User’s experience • Interviews, questionnaires • Various dimensions: control, cooperation, ease of use, efficiency, anxiety, hability etc • Three conditions • No spoken feedback • Limited spoken feedback • Full spoken feedback • 30 subjects

  16. Results • All or nothing • More like a human in the full feedback condition • Before the study subjects wanted to have a timetable for the task, after they preferred a human • No difference in experienced control and cooperation • The speech synthesis better in full feedback mode • Same voice

  17. Open source • Share research results between research groups • Stimulate exchange and co-operation • Language technology readily available for industry • Increased use and insights • The companies do their own prototype development • Robust code • nlpFarm http://sourceforge.net/projects/nlpfarm/

  18. http://sourceforge.net/projects/nlpfarm/

More Related