1 / 12

Multimodal Interfaces

Discover the significance of multimodal interfaces in human-computer interaction, including input and output modalities, sensor fusion, and dual-purpose speech systems. Explore examples and implications.

hclover
Download Presentation

Multimodal Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal Interfaces Scott Klemmer 15 November 2005

  2. Some hci definitions • Multimodal generally refers to an interface that can accept input from two or more combined modes • Multimedia generally refers to an interface that produces output in two or more modes • The vast majority of multimodal systems have been speech + pointing (pen or mouse) input, with graphical (and sometimes voice) output Multimodal Interfaces

  3. Canonical App: Maps • Why are maps so well-suited? • A visual artifact for computation (Hutchins) Multimodal Interfaces

  4. What is an interface • Is it an interface if there’s no method for a user to tell if they’ve done something? • What might an example be? • Is it an interface if there’s no method for explicit user input? • example: health monitoring apps Multimodal Interfaces

  5. Sensor Fusion • multimodal = multiple human channels • sensor fusion = multiple sensor channels • Example app: Tracking people (1 human channel) • might use: RFID + vision + keyboard activity + … • I disagree with the Oviatt paper • Speech + lips is sensor fusion, not multimodality Multimodal Interfaces

  6. What constitutes a modality? • To some extent, it’s a matter of semantics • Is pen a different modality than a mouse? • Are two mice different modalities if one is controlling a gui, and the other controls a tablet-like ui? • Is a captured modality the same as an input modality? • How does the audio notebook fit into this? Multimodal Interfaces

  7. Input modalities • mouse • pen: recognized or unrecognized • speech • non-speech audio • tangible object manipulation • gaze, posture, body-tracking • Each of these experiences has different implementing technologies • e.g., gaze tracking could be laser-based or vision-based Multimodal Interfaces

  8. Output modalities • Visual displays • Raster graphics, Oscilloscope, paper printer, … • Haptics: Force Feedback • Audio • Smell • Taste Multimodal Interfaces

  9. Dual Purpose Speech Multimodal Interfaces

  10. Why multimodal? • Hands busy / eyes busy • Mutual disambiguation • Faster input • “More natural” Multimodal Interfaces

  11. On Anthropomorphism • The multimodal community grew out of the AI and speech communities • Should human communication with computers be as similar as possible to human-human communication? Multimodal Interfaces

  12. Multimodal Software Architectures • OAA, AAA, OOPS Multimodal Interfaces

More Related