1 / 19

Break-out Group # D

Break-out Group # D. Research Issues in Multimodal Interaction. What are the different types. Speech Haptics Gesture Deictic Head and eye movement EEG Electrocephalograms physiological measurements. What has been done so far?. Semantic fusion of information Speech and Gesture

ainsley-barton
Download Presentation

Break-out Group # D

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Break-out Group # D Research Issues in Multimodal Interaction

  2. What are the different types • Speech • Haptics • Gesture • Deictic • Head and eye movement • EEG Electrocephalograms • physiological measurements

  3. What has been done so far? • Semantic fusion of information • Speech and Gesture • Preliminary efforts as to what types of modalities to intergrate. This is application dependent. • Need standardization at the level of devices and types of information to be fused

  4. Open Research Problems: • Should we stay with current paradigms or invent new methodologies? • There is no unifying framework for interaction in terms of devices/semantic integration. This is due to the lack of general purpose application. • We see specific applications eg, simulation, medical training

  5. Open Research Questions • How to deal with specific tasks in terms of fusing channels. How should channels be fused. • How to do transitions between tasks, e.g., manipulation vs loccomotion • Need more experimentation and a theory as to where VR is needed?

  6. Open Research Questions • Formal study of tasks within applications(e.g., manipulation, selection, navigation, changing of attributes, numerical input) • Need more research on output. So far mostly visual and oral.

  7. First breakout group • Taxonomy • Semantics • Cross-modal Representations (actions/perceptions)

  8. Applications/Output Group Second Meeting New Issues we Discussed in the afternoon

  9. DM: Third Breakout Group: Applications/Output • Human Perception of the environment • Integration with Input • Relationships to basic principles

  10. Human Perceptive abilities • Vision Technology: Limitation in terms or lighting or real time rendering • Limitations for other channels: Haptics, audio, olfaction, taste • The type/mix of output depends on the application • This is related to the internal representation

  11. Continued • Issue of using many modalities to offset the limitations of each modality. • Right now we do not have enough research data to support that. • Do we or not need to represent exactly the environment? • Application dependent

  12. Continued • Abstraction vs exact representation • Application dependent • Exact physical simulation vs fake physics. Ok or not to fool the user? • Probably application and technology dependent.

  13. Other Human Perceptive Modalities • Olfaction and taste: very little research • Some modalities are better understood than others (e.g., visual vs haptic or olfaction)

  14. Continued-Summary • Big issues: • Sensory substitution • Level of detail (variable resolution) • Sampled vs synthetic generation • Online or offline computation • Preservation (or not) of individuality e.g two people with different sense of taste or heat etc • Higher-level emotional augmentation

  15. Integration with Input • Haptics is the most widely used output sense that is also used for input • Head orientation, whole-body position, eye gaze also • Some output must be tightly coupled to input (it’s at the physical level) • Head motion to view changes, 3D audio

  16. Integration with Input (cont.) • Eye gaze-based control requires some interpretation • Intentional vs unintentional movement • When is a gesture a gesture?

  17. Relationship to Basic Principles • Mapping semantics to output • One or multiple representations for all modalities eg. Language and visual output where we have a common representation but gets translated differently for output • Spatio-temporal synchronization • Cross-modal representation (actions/perceptions) • Account for individual differences

  18. Future Paper Topics • All the previously mentioned open problems • Short Term • Update of the NRC report on modalities • Medium-Term • Modeling, Coupling and Output of modalities • In particular model smell and taste

  19. Future Paper Topics • Long Term • Further modeling and coupling • Advanced display technology • Personalization of output

More Related