190 likes | 310 Views
Multimodal Information Exchange and Dynamic Adaptation. Nadine Sarter Thomas Ferris Shameem Hameed University of Michigan. Multimodal Adaptive Displays. Future battlefield operations will be highly complex and dynamic and require effective information systems/exchange Our approach:
E N D
Multimodal Information Exchange and Dynamic Adaptation Nadine Sarter Thomas Ferris Shameem Hameed University of Michigan
Multimodal Adaptive Displays • Future battlefield operations will be highly complex and dynamic and require effective information systems/exchange • Our approach: • Multimodal displays (including vision, audition, and touch) • Context-sensitive hybrid adaptive/adaptable information presentation
Research Activities • Created flexible computer-based simulation platform that supports co-located and remote synchronous collaboration • Used platform for series of studies on • Natural patterns and preferences of modality usage • Preattentive monitoring of mission health • Crossmodal spatial and temporal links in attention
Natural Patterns of Modality Usage • People do not necessarily interact multimodally just because a multimodal interface is made available • Multimodal interaction primarily in the context of spatial tasks and to support complementarity • Switch modalities mostly for the purpose of recovering from communication breakdowns • In the context of human-human interaction, most modality combinations were sequential in nature (“contrastive functionality”) • Modality usage patterns evolve as a function of team coordination and change in response to factors such as scenario demands, the mission phase, and group dynamics.
Crossmodal Links in Attention • Crossmodal spatial and temporal links • The modality & location of a stimulus in one modality may facilitate/hinder processing of subsequent stimulus in different modality • Effects manifest only within a certain time interval between stimuli (SOA – Stimulus Onset Asynchrony) • Related information should be co-located… (the binding “problem”)
Previous XSL Studies • Spartan laboratory environments • Simple and artificial cues and tasks • Do we see these effects in more complex environments with real-world tasks and stimuli? SOA e.g., Spence & Driver, 1997 ~100-300ms
Experimental Setup Periscope Display Thermal detection system display FBCB2 shared map display Satellite data uplink display Push-to-talk radio button strapped to index finger Left speaker Right speaker Remote eye-tracking camera Joystick for UAV Tactors worn on wrists Tactors strapped to outsides of thighs
Method • 12 cadets and 3 graduates from the University of Michigan Army ROTC program (7 females and 8 males) • Each participant played Stryker vehicle commander (VC) for the first of a convoy of vehicles in a simulated night-time rendezvous mission • Throughout the mission, participants were presented with 48 visual, auditory, or tactile targets, either in isolation (‘uncued’ trials, n=24) or preceded (various SOAs) by an ipsilateral (same-side, n=12) or contralateral (opposite-side, n=12) peripheral cue in a different modality
Main Findings • Confirm that crossmodal spatial links affect performance in more complex settings • Cuing effects were larger and response times longer and varied to a higher degree than in earlier research • Crossmodal asymmetries: • Ipsilateral crossmodal cuing was beneficial only for auditory cuing of visual targets but not vice versa • Significantly faster responses for contralateral tactile cuing of auditory targets but not vice versa • Visual-tactile cue-target combinations showed a similar trend favoring contralateral presentations • Faster response times for contralateral presentations in some cases may be the result of IOR
Cueing of Visual Target IOR???
Main Findings • Confirm that crossmodal spatial links affect performance in more complex settings • Cuing effects were larger and response times longer and varied to a higher degree than in earlier research • Crossmodal asymmetries: • Ipsilateral crossmodal cuing was beneficial only for auditory cuing of visual targets but not vice versa • Significantly faster responses for contralateral tactile cuing of auditory targets but not vice versa • Visual-tactile cue-target combinations showed a similar trend favoring contralateral presentations • Faster response times for contralateral presentations in some cases may be the result of IOR
Context-Sensitive Display Design • Hybrid approach – “Delegation” • Combine positive aspects of adaptive (system-initiated) & adaptable (user-controlled; management-by-exception) interfaces • Combine/negotiate among multitude of drivers related to operator, cues, and environment
Mental Workload • Heart rate • Confounded by physical component & mental stress • Heart rate variability • Variation in time interval between consecutive heart beats • Relatively stable index of mental workload • Shows shifts from rest state to task state • Shows varying levels of workload in task state • Power spectrum analysis (Workload) From Rowe et al, 1998
Availability/Appropriateness of Modality • Modality availability • Modality may have become temporarily or permanently unavailable due to ambient conditions/events (e.g., explosion/ambush) • Modality appropriateness • Nature and type of information conveyed • Certain modalities more appropriate than others for certain types of information • e.g., spatial information (geographic location) is best conveyed visually
Challenges • Weight assignments for modalities may need to be adjusted • Deadlock arbitration module • Start with “hard” constraints (detectability, availability) • Then, choose modality with high appropriateness index • If all those are same, consider previous cue modality and timing
Experiment – Final Step • Simulation and hybrid interface have been implemented • Experiment is designed and under IRB review – will be conducted in Fall • Will examine feasibility and effectiveness of the approach and compare to “static” information presentation
Tactons Structured, complex tactile signals which communicate abstract messages (Brewster & Brown, 2004) Braille Text Tactons Visual icons
Tactile “Cues” vs. Tactons Tactile “Cues” Tactons • One or few parameters • modulated • Literal, critical to maintain • intuitive mapping • Very limited information • content • Usually multiple parameters • modulated • Abstract patterns represent • larger concepts or messages • Conscious processing required • to decode message Examples: Interruption management, patient monitoring
Conclusion • Much more complex picture emerges for effective multimodal information presentation • Requiring careful choice of modality pairings, location, timing • Calling for context-sensitive presentation of information • Power of tactons far from being exploited • Beware of guidelines (“adapt modalities to user preferences”) but see, for example, Jones and Sarter (2009) for guidance…