1 / 36

Adaptive Automation for Human Performance in Large-Scale Networked Systems Research Team:

Adaptive Automation for Human Performance in Large-Scale Networked Systems Research Team: George Mason University Raja Parasuraman Tyler Shaw Ewart de Visser Amira Mohammed-Ameen Andre Garcia. Cornell. MIT. GMU. Pitt. CMU Robotics. CMU Psychology. MIT. Scaling of cognitive

lillian-hughes
Download Presentation

Adaptive Automation for Human Performance in Large-Scale Networked Systems Research Team:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Adaptive Automation for Human Performance in Large-Scale Networked Systems Research Team: George Mason University Raja Parasuraman Tyler Shaw Ewart de Visser Amira Mohammed-Ameen Andre Garcia

  2. Cornell MIT GMU Pitt CMU Robotics CMU Psychology MIT Scaling of cognitive performance and workload Level 1,3 Level 1,2 Level 1 Level 2 Level 1-2.5 Level 1-3 Level 1 Level 1,3 Task allocation among humans/agents Probabilistic models of human decision-making in network situations Level 1,2 Level 1-2.5 Level 1,2 Level 3 ? Level 2 Level 1-3 Decentralized control search and planning Level 1,2 Level 2 Information fusion Level 1,2 Level 1,3, 4 Network performance as a function of topology Level 4 Level 2 Communication, evolution, language Level 3 Level 2, 3 Adaptive automation Level 1,2 Level 1

  3. Research Goals Develop validated theories and techniques to predict behavior of large-scale, networked human-machine systems involving unmanned vehicles Model human decision making efficiency in such networked systems Investigate the efficacy of adaptive automation to enhance human-system performance 3

  4. 1st Year Studies • Modeling Decision Making under High Cognitive Load • Complexity: Numbers of Enemy Assets and Messages • Verification Behavior in Networked Systems (in progress) • Effects of Message Pedigree and Training • Adaptive Automation to Support Human Supervision of multiple UAVs • Complexity: Number and types of UAVs • Playbook Interface vs. Scripts vs. Tools • Neuroergonomics-Based Adaptive Automation (in progress) • Transcranial Doppler Sonography (TCD), Cerebral Blood Flow and Cognitive Load First Year Review October 2, 2009

  5. Cognitive Load Varies with N(from Mike Lewis, U Pitt) O(>n) • Our studies focus on O(n) and O(>n) cases • Cognitive limit varies with working memory capacity W O(n) Cognitive limit = f(W) O(1) N of Robots

  6. Study 1 • Hypothesis: Cognitive load limit on operator in O(n) case is predictable from • Number of enemy asset targets • Message complexity • Individual working memory capacity, W?

  7. Add picture Information Panel Neutral Zone Approximate attack range Teammate zone Enemy asset Neutral asset Message Center DDD STUDY 1

  8. Study 1 DDD 4.0 Simulation Destroy as many enemy assets (single or squadrons) as possible Prevent enemy asset incursion into neutral zone Communicate target information to teammate (Neutral assets can also turn into enemy targets) Two levels of enemy cognitive load (low, high) number of enemy asset incursions Speed of enemy asset incursion Network messages from agent provide operator with critical information to achieve mission success No messages Noisy messages (only 20% of the messages directly relevant to mission objectives (e.g. destroy enemies) Direct information (all messages directly relevant to mission objectives)

  9. Study 1 Methods • Participants • 30 adults (18 men, 12 women) aged 18-26 years, 15 were ROTC cadets • Dependent measures • proportion of enemy asset incursions into neutral zone • number of enemy assets destroyed • time to destroy enemy targets • number of messages acknowledged • quality of messages passed to team-mate • Overall performance score • red-zone incursions: - 2 points/second • destroy enemy asset: + 100 points • enemy attack on own asset: - 100 points • friendly fire: - 50 points First Year Review October 2, 2009

  10. Study 1 Results: Red Zone Performance First Year Review October 2, 2009

  11. Study 1 Results: Overall Performance First Year Review October 2, 2009

  12. Study 1 Results • Simple linear modeling of • Red zone performance • Overall mission score (combining all performance measures) • Performance = w1 + w2ne + w3p(m) +  • ne = number of enemy assets • p(m) = proportion of relevant messages • w1,w2,w3 = weights •  = error • Variance accounted for: • Red zone performance: 46% • Overall mission score: 42%

  13. Study 1 Results • Unexpected finding • Very high inter-individual variability in performance • Range of scores: • red zone incursion proportion: 0 to 80% • overall points: -2994 to 2600 • Working memory capacity WM • Individual differences in Operation span measure of working memory • w1 + w2ne + w3p(m) + w4WM + • Variance accounted for with: • Red zone performance: 76% • Overall mission score: 62%

  14. Follow-Up to Study 1Collaboration with Cornell Probabilistic modeling of human decision-making performance Compare model to simple linear model Identify and quantify human “cognitive bottlenecks” Identify points for “adaptive tasking” or adaptive automation Scale up to larger networks (more UVs and agents) 14

  15. Study 2 (in progress) • Problem: Human operator under-trust (skepticism) and over-trust (complacency) can limit usefulness of information sources in large networked systems • Hypothesis: Trust and complacency in networked systems can be indexed by verification behavior (information sampling) • Number and complexity of messages • Message pedigree (hub vs. isolated node in network)

  16. Study 2 Forest firefighting simulation Operator services UAV requests to combat different types of forest fires with unique requirements Assets: 2 (simple) or 5 (complex) UAV types Operator receives messages from network requesting service Automated agent recommends decision choice Operator can verify UAV and fire information parameters Source of information: Base HQ (hub node) or team-mate (isolated node)

  17. Training Manipulation Two 10 minute practice sessions (one for each UAV complexity level) in which 20 forest fires need to be serviced Information group Told there will be unreliable messages, but experience none. Told that messages from Base HQ more reliable than from isolated node Experience group Base HQ: 90% reliable (9 of 10 messages give correct recommendation) Isolated node: 60% reliable (6 of 10 messages give correct recommendation)

  18. Complex scenario (5 UAV types) 1 1. Dumps Fire Retardant 1 1. Photographs with Infra-red 1 1. Photographs with TV 2. Carry smoke jumpers 1. Photographs with TV 1 2. Picks up victims 1 1. Photographs with TV

  19. Operator receives automated dispatch requests. FROM ISOLATED NODE: SEND 1 BLUE FROM BASE HQ: SEND 1 BLUE Click to Acknowledge Notice time Read Recommendation & Decide Select Assets to be sent and click OK 21 November 2014

  20. Trust…. but Verify Operator can verify the automated agent recommendation by clicking “Display information.” 21 November 2014

  21. Study 2 Results Full verification level Optimal verification “Skeptical” “Complacent” First Year Review October 2, 2009

  22. Study 3 • Hypothesis: Playbook adaptive automation enhances decision-making performance in a realistic, near-term [O(n), with n=3] simulation of heterogeneous human-UAV teams

  23. Study 3 • Examine efficacy of adaptive automation — specifically the Playbook for supervisory control of multiple heterogeneous UAVs • Compare three Levels of Automation • Tools • Scripts • Playbook First Year Review October 2, 2009

  24. Title Multi-UAV Simulation 3 Fixed-Wing UAVs Map Display with Named Areas of Interest (NAIs) Sensor Display Multi-Function Display • UAV capabilities • Alpha (blue) • paint civilian targets and • AutoTrack Targets • Bravo (pink) • 1) civilian targets, • 2) AutoTrack Targets • laze weaponized* targets (cannot prosecute) • Charlie (orange) • paint civilian targets • AutoTrack • Prosecute weaponized targets (cannot lase) First Year Review October 2, 2009

  25. Tools (Low LOA) Manually move waypoints to position UAVs to monitor NAI. Manually paint civilian targets Autotrack humvees Manually lase weaponized targets Manually prosecute weaponized targets MUSIM STUDY 1

  26. Scripts (Intermediate LOA) Automatically task individual UAV’s to monitor 1 or 2 NAI’s Manually paint civilian targets Autotrack humvees Automatically lase weaponized targets. Manually prosecute weaponized targets. MUSIM STUDY 1

  27. Plays (Flexible LOA) Automatically assign ALL UAV’s to monitor ALL NAI’s Manually paint civilian targets Autotrack humvees Reconfigure team so that 2 UAVs monitor 3 NAIs while the other UAV tracks a target. Simultaneously lase and prosecute MUSIM STUDY 1

  28. 3 LOAs Scripts Tools Plays MUSIM STUDY 1

  29. Study 3 Results • Plays reduced target acquisition and prosecution times • Plays also reduced overall mental workload

  30. Study 4 (in progress) • Hypothesis: Cognitive load limit on operator in O(n) case can be assessed dynamically by measuring cerebral blood flow with Transcranial Doppler Sonography (TCD) • TCD measurement of operator cognitive load could be used to drive adaptive automation to support the operator

  31. Transcranial Doppler Sonography (TCD) and Cerebral Blood Flow

  32. Study 4 Methods • Participants • 16 adults aged 18-30 (to date) • Air defense task (DDD 4.0, as in Study 1) • Task load (number of enemy assets) increased unpredictably from low to high in the middle of mission • No messages or messages present conditions • Cerebral blood flow velocity measured in left and right hemispheres of the brain First Year Review October 2, 2009

  33. Point of transition

  34. Results to Date Carryover effect of cognitive load? Transition

  35. Publications Grier, R. A., Parasuraman, R., Entin, E. E., Bailey, N., & Stelzer, E. (2008). A test of intra- versus inter-modality interference as a function of time pressure in a warfighting simulation. In Proceedings of the Human Factors and Ergonomics Society. Santa Monica, CA, pp, 1229-1232. Parasuraman, R. (2009). Neuroergonomics applied to adaptive automation for supervision of multiple unmanned vehicles. In Proceedings of the Army Science Confererence, Orlando, FL. Parasuraman, R., Cosenzo, K., & de Visser, E. (2009). Adaptive automation for human supervision of multiple uninhabited vehicles: Effects on change detection, situation awareness, and mental workload. Military Psychology, 21.270-297. Parasuraman, R., de Visser, E., & Shaw, T. (2009). Individualized adaptive automation. In Proceedings of the Human Factors and Ergonomics Society Conference. San Antonio, TX.

  36. Planned Studies • Extend DDD study to 2-person teams • Extend modeling of decision-making performance performance to 2-person teams • Extend network message verification study to examine effects of cost of verification, message reliability, and network size and topology • Examine feasibility of implementing adaptive automation to mitigate cognitive load based on measurement of cerebral blood flow First Year Review October 2, 2009

More Related