1 / 30

Multimodal Interaction

Multimodal Interaction. Modalities vs Media. Modalities are ways of encoding information e.g. graphics Media are instantiations of modalities e.g. a particular image. How Do Multimodal Systems Differ?. Domain/application Available media Modeling of context/environment Modeling of user

orleans
Download Presentation

Multimodal Interaction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal Interaction

  2. Modalities vs Media • Modalities are ways of encoding information • e.g. graphics • Media are instantiations of modalities • e.g. a particular image

  3. How Do Multimodal Systems Differ? • Domain/application • Available media • Modeling of context/environment • Modeling of user • Focus of research

  4. Example Multimodal Systems • Not speech-centric • MIT paintbrush, soundbrush • http://www.youtube.com/watch?v=04v_v1gnyO8 • http://www.youtube.com/watch?v=iZbe3t8YSf4 • http://www.youtube.com/watch?v=18RY8Jgid20 • Wearables • http://www.gatech.edu/innovations/wearable/

  5. Example Multimodal Systems • Speech-centric • MSOIP • COMIC • SmartKom

  6. MSOIP Keywords • Multimodal mobile dialog • Integration of speech and pen input • User modeling for presentations Johnston et al. 2001

  7. MATCH Video http://www.research.att.com/~johnston/ Scroll down to the bottom of the page

  8. About MATCH • What input modalities? • What output modalities? • What application(s)? • What aspects of context?

  9. COMIC Keywords • Ambient intelligence • HHI/HCI research • Collaborative problem solving • User modeling • Avatar Alexandersson et al. 2004

  10. COMIC Video http://www.hcrc.ed.ac.uk/comic/demos/facial-animation/

  11. COMIC Video http://www.hcrc.ed.ac.uk/comic/demos/slot/

  12. About COMIC • What input modalities? • What output modalities? • What applications? • What aspects of context?

  13. SmartKom Keywords • Multimodal dialog across • applications • devices and • situations • Avatar • Situation aware Alexandersson et al., Reithinger et al. 2003

  14. SmartKom Video http://www.smartkom.org/start_en.html I showed the SK-Mobile one, but the other one is also interesting.

  15. About SmartKom • What input modalities? • What output modalities? • What applications? • What aspects of context?

  16. Parts of a Multimodal System Text Out Text In Speech Out Speech In Present Out Gesture In Interpreter Generator Dialog Manager Knowledge Base

  17. HCI and Multimodal Systems • Input integration/fusion • Representations • Effective help • Quality presentations • Managing context • Understanding the user

  18. Different Uses of Modalities • Concurrent or sequential • Redundant or • Complementary or • Contradicting

  19. Input Integration/Fusion • Key elements: • Time • Multiple uses of some modalities • Error rates • Typical approach is to map straight to semantics if possible

  20. Representation • Increasing use of XML-based languages (SMIL, EMMA) • But these don’t solve the semantic problems • Keep ‘backbone’ knowledge separate from ‘peripheral’ information (Alexandersson et al.)

  21. Effective Help • How do each of the systems provide the user with: • Explicit help? • Implicit help?

  22. Quality Presentations • Talking heads • Advantages • Disadvantages • Informative presentations are key • User modeling/adaptive presentations are a bonus • These systems go beyond scripts

  23. Managing Context • What kinds of context are there in a mobile multimodal interaction?

  24. Understanding the User • What kinds of information can we gather about users in general? • About one user in particular? • How can we use this information?

  25. Commercial Multimodal Systems • Most are for research • Military • Training and battlefield • Education • Tutoring systems • Commercial ones include: • Wii: http://www.youtube.com/watch?v=n4nZVAEeitU • Microsoft surface: http://www.youtube.com/watch?v=rP5y7yp06n0

  26. TradeOffs • You get: • More intuitive technology • More information, more easily • Less (dumb stuff) for you to do • You trade: • Privacy • Control

  27. Towards the Future • Design • Multimodal systems in virtual worlds, or crossing over from virtual to real worlds • Ambient multimodal interaction • Implementation • Mashups – user controlled • Pervasive multimedia

  28. Towards the Future • http://www.youtube.com/watch?v=FMJwURqpFWs • http://www.programmableweb.com/mashups

  29. SciFi? • Lathe of Heaven by Ursula LeGuin • Summa Technologiae by Stanislaw Lem • Fast Times at Fairmont High by Vernor Vinge • The Human Machine Merger, talk by Raymond Kurzweil (at http://www.kurzweilai.net/meme/frame.html?main=memelist.html?m=6%23581)

  30. Additional Info • http://search.techrepublic.com.com/search/multimodal+system.html • http://www.w3.org/2002/mmi/

More Related