1 / 33

Multimodal Design & Technologies

Multimodal Design & Technologies. Sidney Fels. Overview. Introduction Human I/O: Interface Examples Bringing Modalities Together Summary. Introduction. Two messages knowledge of modalities motivates design modalities can complement each other challenges and pitfalls

braith
Download Presentation

Multimodal Design & Technologies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal Design & Technologies Sidney Fels

  2. Overview • Introduction • Human I/O: Interface Examples • Bringing Modalities Together • Summary

  3. Introduction • Two messages • knowledge of modalities motivates design • modalities can complement each other • challenges and pitfalls • Applications in virtual environments • communication of experience • intimacy and embodiment

  4. Human Information Processing • Input • 5 usual senses • vision, hearing, touch, taste, smell • position and motion sensing systems • Output • intentional • neuromuscular, movable, verbal • non-intentional / biopotentials • galvanic skin response (GSR), heart rate, brain, muscle activation • Cognitive • memory, decision making, tracking, learning

  5. Visual Display Technologies • examples of designs exploiting human capabilities • Virtual Retinal Display (VRD) • Cubby • CAVE Automatic Virtual Environment

  6. Virtual Retinal Delay (VRD) From HIT lab, U. Of Washington (Furness et al., 1991) Microvision continuing work

  7. Cubby (Djajadiningrat and Gribnau, 2000)

  8. CAVE Illustration (U. of Illinois, 1992)

  9. Haptics Display Technologies • examples of touch and force feedback • Pantograph • Phantom • CyberForce

  10. Pantograph (Hayward, McGill) • 2 dof translation - 10cm X 10cm • FF/B mouse • acceleration best for shock and hard contacts

  11. Phantom (Massey, SensAble Devices) • thimble on finger • single point force feedback • 3 degrees of freedom

  12. CyberForce (Kramer, Immersion Corp.) • Exoskelton • arm forces • Tendons • grasp forces

  13. tricks: visual / aural illusionsexamples of metacognitive gap • visual: never show the point penetrating the surface, even if it is • aural: play a crisp contact sound on contact this makes the surfaceappear stiffer/harder(very robust effect) actual: displayed: BUT: if time offset too great, opposite effect

  14. Other Haptic Devices • Tactile feedback • Temperature • Sheer forces

  15. challenges for haptic interaction design • continuous vs. discrete manual control • displaying interaction potential • embedding haptic interfaces • ensuring tight sensory coupling • interacting with other modalities

  16. Olfactory Devices • How do we smell? • different theories: • i.e. chemical, infrared absorption • different perceptual mappings: • smell prism • four odors: fragrant, acrid, burnt and caprylic • domain specific: wine, beer, etc. • many, many others... • Acuity is great - 10,000 times more sensitive than taste • negative adaptation occurs • you get used to the smell

  17. Olfactory Interfaces • Smell camera • Susnick & Raknow, Nature, 2000, 406, 710-714. • Smell display • liquids, gels, microencapsulation • Application Research: • Joe Kaye: • inStink, Dollars & Scents, Scent Reminder, and more... • Fels, Gauthier, Smith: Interactive Yoga system • Smell Display Research and Products: • DigiScents (bankrupt), TriSenx (bankrupt) • DIVEpak (Southwest Research Institute, 1993) • see notes

  18. Olfaction Display Challenges • smell synthesis • many smells vs. mixing base smells • control breathing space • sealed room with air filtration • air control in front of and behind user • sealed pod • tethered mask • tubes into an HMD from pack • built into HMD

  19. Taste • taste buds for • sensations of sour, salty, bitter, sweet and umami • receptors not completely resolved • umami receptor (Zucker et al., 2002) • extremely complex and poorly understood • interacts with olfaction • display = food?

  20. Biopotentials: Examples • Galvanic Skin Response (GSR) • Affective Computing, i.e. Galvactinator, Scheirer and Picard • Heart Rate (HR) • 2 Hearts Musical System, (McCaig and Fels, 2002) • Brain activity (EEG) • Brain Computer Interface (BCI) • Muscle activity (EMG)

  21. Summary of Human I/O • Multitude of input/output systems • all active at once • I/O mechanisms usually depend upon • cognitive context • emotional contexts • All these systems available for applications • complement each other • Multimodal design looks at: • integration • substitution • complement

  22. Multimodal Design: Bringing it Together • User centred and non-user centred • Intimacy and Embodiment • automatic behaviour • sources of aesthetics

  23. Intimacy and Embodiment • Want interfaces that feel “good” to use • Humans and machines intimately linked • degree of intimacy supported may determine success • Types of relationships: • human to human • human to machine

  24. Intimacy • Intimacy is a measure of subjective match between the behaviour of an object and the control of that object. • extension of “control intimacy” from electronic musical instruments analysis (Moore, 1997) • High intimacy implies: • object feels like an extension of self • satisfaction derives from interacting with object • emotional expression flows • requires cognitive effort to prevent

  25. Intimacy, Embodiment and Aesthetics Aesthetic self object Result Case 1: Object disembodied from Self Case 2: Self embodies Object self object Control Case 3: Self disembodied from Object Reflection object self Case 4: Object embodies Self object Belonging self

  26. Intimacy and Embodiment Design Examples • Many excellent examples out there of interesting designs • Many at Siggraph: • Jam-o-Drum (Blaine et al.), Wooden Mirror (Rozin), etc. • Tangible Bits work (Ishii et al.) • Ubicomp (Weiser and more) • Wearable computing (Mann and others) • Art, Entertainment • Virtual reality

  27. Application: Iamascope (Fels and Mase, 1997)

  28. Iamascope Overview Video

  29. FlowField: Semantics of Caress(Chen, Fels and Min, 2003) • Whole hand interaction in VR • use multi-touch sensitive pad • immersive display • Idea: • allow users direct manipulation of fluid • use particle simulation for fluid • aesthetics was important • hand manipulation on hard surface mapped to obstructions in the flow field

  30. FlowField: Semantics of Caress

  31. Summary • Multimodal interfaces need to consider • human information processing • matching interface to task • use complementary modes where appropriate • intimacy and embodiment • Plenty of research opportunities

  32. Thank-you! Please fill out your comments forms. www.icics.ubc.ca/hci/multimodal/

More Related