The Man-Machine Integration Design & Analysis System (MIDAS): Recent Improvements

1. The Man-Machine Integration Design & Analysis System (MIDAS): Recent Improvements Sandra G. Hart Brian F. Gore Peter A. Jarvis NASA Ames Research Center Moffett Field, CA 94035 Sandra.G.Hart@NASA.gov/650 604 6072 10/19/04

2. Outline • Human Performance Modeling • MIDAS Phase 1: Initial design • Early applications • MIDAS Phase 2: Move from Lisp to C++ • Recent applications • MIDAS Phase 3: PC Port/Integrate Apex

3. Human Performance Models: Components Psychological Models Timeline Sensory Models Task Network Anthropometric Models Performance: WL Biodynamic Models Performance: Time Model Architecture, Library, Tools Team/Org Models Performance: SA Vehicle Models Performance: Errors Equipment Models Visualization Environment Models FoV/Reach Envelope Procedural Models

4. Human Performance Models: Architectures Anthropometric/Biodynamic: Physical characteristics of human body; static & dynamic; population characteristics; limitations [RAMSIS, JACK] • Task network: Top-down, based on sequences of human/system tasks (derived from task analysis Psychological theories, mathematical models, descriptive functions Cognitive: Bottom-up, combine theory-based models of memory, decision making, perception, attention, movement, etc Vision: Computational representation of the way the human visual system processes an image to predict performance given image characteristics

5. Human Performance Models can… • Generate hardware, software, training requirements for tasks that will involve human operators • Depict operators performing tasks in prototype workspaces and/or in remote or risky environments • Perform tradeoff analyses among alternative designs and candidate procedures, saving time and money • Identify general human/system vulnerabilities to estimate overall system performance and reliability • Provide dynamic, animated examples for trainingand developers • Generate realistic schedules and procedures

6. Phase 1

7. Data Input Overview A comprehensive suite of computational tools - - 3D rapid prototyping, models of perception, cognition, response, real- and fast-time simulation, performance analysis, visualization - - for designing and analyzing human/machine systems was developed primarily in Lisp on a fleet of SGIs Data Analysis Run Time Visualization

8. Features • Pioneered the development of an engineering design environment with integrated tools for rapid prototyping, visualization, simulation and analysis • Advanced the capabilities and use of computational representations of human performance in design including a state of the art anthropometric model (Jack®) • Flexible enough to support a range of potential users and target applications But…. • Component models written in Lisp, Fortran, C, C++ • Required a suite of SGI machines • Modeled a single operator • Time based rather than event based; scheduler established optimal inter-leaving of task components • No emergent behaviors

9. Richmond, CA Police: 911 Dispatch • Goal: Upgrade the facilities and procedures used in the 911 dispatch facility • Accomplished: • Modeled control console and dispatch activities in MIDAS • Evaluated prototype graphical decision aid

10. US Army Air Warrior • Goal: Establish baseline performance measures for crews flying Longbow Apache with and without MOPP gear • Accomplished: • Modeled copilot/gunner with Jack® (95th male <> 5th female) • Rendered cockpit using CAD files from manufacturer • Simulated performance of more than 400 activities • Measured reach, FoV, workload, timelines

11. Short Haul/Civil Tiltrotor • Goal: Evaluate crew performance/workload issues for steep (9º), noise abatement approaches into a vertiport • Accomplished: • Constructed MIDAS models of normal and aborted approaches • Contrasted impact of manual vs automated nacelle control modes

12. NASA Shuttle Upgrade • Goal: Support development of an advanced orbiter cockpit with an improved display/control design • Accomplished: • Created virtual rendition of current shuttle cockpit • Conducted simulation of first 8 min of nominal ascent • Provided quantitative measures of workload/SA, timing

13. Phase 2

14. Features • Decreased model development from months to weeks • Increased run-time efficiency from 50x RT to near RT • Multiple operators • Modeled external vision, audition, situation awareness • Conditional behaviors emerging from interaction of top-down goals and environmentally driven contexts • Option of non-proprietary “head & hands” model But… • The interface still user un-friendly • SGI platform • Cognitive models no longer state of the art • Performance moderating functions not integrated

15. Overview of Architecture

16. User Interface • The interactive graphical user interface is used to create models, specify and run simulations, and view data. It is organized into a hierarchical series of screens or editors that are navigated with tabs • Different views of the simulation are offered: Structure, Geometry, Outline, Animation, Real-time/post-hoc data

17. Vehicle Models • A modeled vehicle represents the combination of a guidance/ dynamics model and a visual representation • The guidance/dynamics model moves the vehicle along a prescribed route. MIDAS provides two: • NoE helicopter model • Simple point mass model (used to model arbitrary vehicles in a generic way) • The visual representation is CAD geometry chosen from the MIDAS library or developed by the modeler.

18. Environment Model • Tools are provided to model the environment outside the crew station (e.g., terrain, weather, etc) • Terrain is modeled as a single object • Features are simple objects that have no inherent behavior and do not move • One weather condition may be applied to the environment by specifying lighting/visibility (these are used by the visual perception model)

19. Crew Station/Equipment Models • The “crew station” is a collection of equipment with which operators interact • Crew station models may be given a graphical representation for animation • Multiple crew stations per vehicle and multiple operators per crew station possible

20. Anthropometric Models • Anthropometric models provide an animated, 3D graphical representation of one or more modeled human operators for visualization • Jack ® (developed at U Penn/distributed by UGS): full-body figure & realistic movements • Head and Hands model: government-developed representation adequate for many purposes for users without a Jack license

21. Vision Models • Visual attention modeled as single “cone”, varying from 3-15º based on task type. • External vision: • Peripheral: 160 degrees • Foveal: 2.5 degrees • Perceivability: f(visibility, size, distance, local contrast ratio) • Perception level: f(dwell time, perceivability) • Levels of Perception: • Detection • Recognition • Identification • Internal vision: • Symbolic (check read) • Digital (exact value) • Text (character string)

22. Auditory Model • Only within crew station • External sounds are represented only if channeled through equipment • Two Stages of Processing: • Detection • Comprehension • Content: • Verbal strings • Signals • All or none processing (Interruptions disrupt entire message)

23. Symbolic Operator Models • Significant advance over earlier version, which required specification of all activities at primitive task level • High-level scripting language, Operator Procedure Language (OPL) serves as front-end to a reactive planning system (RAPS) • User-supplied procedures are instructions for accomplishing tasks • Manages knowledge and beliefs, integrates human actions with scenario events

24. Memory Model • Simpler model than in MIDAS 1.0 • Distinction between long-term/short-term memory was lost • Memories are represented as a database of assertions or beliefs that are symbolic expressions describing the property of objects • Memory can be examined by powerful tools in a querying language built into OPL

25. STAGES CENTRAL PROCESSING ENCODING RESPONDING SPATIAL MANUAL RESPONSES VISUAL VERBAL VOCAL MODALITIES AUDITORY SPATIAL CODES VERBAL Attention Model • Based on Wicken’s Multiple Resource Model. • Acts as a mediator that maintains an account of attention resources in six different “channels” • Necessary attention resources must be available before primitive tasks are initiated • Task onset may be delayed if insufficient resources

26. Output Behavior Models • If required resources are available an activity that corresponds to a primitive procedure is created • Physical actions and their effects on equipment or environmental objects are modeled, regulated by a motor control process • 60+ primitive tasks are available in a Procedure Library with pre-defined load values; easy to add more

27. Simulation System • Engine/executive (uses discrete-event, fixed-time increment approach for advancing the simulation) • Data collection mechanisms for generating runtime data that is graphically displayed which the simulation runs and is saved for post-run analyses • Event generation mechanism provides a way for timed events to occur on schedule or with stochastic variance • Provisioning system allows users to change the simulation and re-run without re-loading/re-starting

28. Workload & Situation Awareness • Workload calculations based on McCracken & Aldrich (1988) • Load levels for Visual, Auditory, Cognitive, and Psychomotor dimensions are defined for task primitives on a scale of 1-7 • Momentary load based on aggregation • SA calculations based on: • Ratio of operator’s relevant knowledge/required knowledge • Distinguishes actual SA from perceived SA Situational elements can be objects in the crew station or the environment that define a “situation” or are in the operator’s memory and are operationally relevant. • WL and SA values offer a powerful way to simulate realistic errors

29. Validation: Search & Rescue Mission

30. Nominal baseline approach/landing and late runway reassignment (sidestep) with and without SVS display 1000' Lineup on Final ATC-Commanded Runway Sidestep 850' Ceiling 650' Missed Approach Parallel Runways Comparison of Models to Simulator Data ACT-R/PM U of Illinois Rice University D-OMAR BBN Technologies Air MIDAS San Jose State University IMPRINT/ACT-R MAAD & CarnegieMellon A - SA U of Illinois Goodness-of-fit of individual model outputs to empirical data Preliminary timeline, SA, attention, wkld, analysis,task execution times error vulnerabilities MIDAS NASA-ARC/Army

31. Nominal Approach & Landing Simulation • PF scanning for TFX, runway • PNF monitoring PFD, Nav • PF/PNF monitoring radio • Flaps 30º/set & confirm • PF requests before landing checklist • PNF checks/responds hear down • PF confirms visually/verbally • PNF checks/responds flaps 30 • PF confirms visually/verbally • PNF checks/responds speed brakes set • PF confirms visually/verbally • PNF declares checklist complete • PF sets/declares DA at 650 • PNF visually confirms DA set • Note passing FAF • Confirms final descent initiated

32. Traffic Call During Approach • Final approach checklist is complete • ATC call with traffic advisory • Both pilots scan for traffic “I don’t see it” • Neither pilot notices as the decision altitude is passed • After the fact, the First Officer notices: “We’re past FAF and not descending” • Crew must decide whether to continue with the approach or abort

33. Life Sciences Glove Box Virtual Glovebox • Challenges: • Astronauts must follow detailed instructions within strict time constraints; failure to do so introduces risk of science mission failure Role of Computational Modeling • Predict interactive influences of microgravity (posture, bracing, precise movements, placing, moving, stowing) to develop/evaluate procedures • Watching an animated dry run enables efficient communication among scientists, implementers, astronauts; more effective training Life Sciences Glovebox Payload Development Unit received at Ames from the National Aerospace Development Agency of Japan (NASDA) Onboard KC-135 MIDAS rendering

34. Life Sciences Glove Box Simulation • Goal: Predict astronauts’ performance of complex experiments designed to answer questions about living organisms’ adaptation to the space environment • Objectives:Evaluate feasibility of following proposed procedures within time/performance constraints; ID factors that will increase risk of mission failure [e.g., waiting too long to photograph slides; interruptions; task requires (unavailable) resource(s)] • The Task: • Turn on experimental equipment (monitor, microscope, camera) • Measure cell density/viability for each of 6 samples • Invert sample vial • Place aliquot of sample on slide • Place drop of viability stain in sample • Record time on sample record • Place cover slip on slide • Observe on microscope • Take photographs within specific time window • Dispose of trash, return vials to containers, turn equipment off

35. Cell Staining/Photographing Experiment

36. Phase 3

37. MIDAS v3.0 Features • Runs on high-end PC • Simple model of microgravity influence on performance • Physics model of microgravity impact on objects available • Simple within-task fatigue model implemented • Fatigue state model (U Penn/Astronaut Scheduling Assistant) selected • Notion of task duration - - how long a task should take as well as how long it did take • Grasping, moving, manipulating objects in workspace • Apex will become the heart of the Task Manager and enable multi-tasking, task prioritization, shedding, deferral, resumption • Task primitive definitions include failure modes (time/quality) that enable the occurrence of emergent behaviors • Mission success/performance measures computed: vulnerability to error, slipped schedules; performance degradation

38. MIDAS v3.0 Structure Task Manager Plans Monitors Remembers Senses Actuates Task Network List of Tasks/Procedures Commands Results Timeline Mission Environment Physical Simulation Perceives Attends Moves/Acts Changes Workstation Geometry Fit/Reach/Vis envelope Dynamic models Task executions Dynamic Animation Library Primitive tasks Human model Task state Operator state Mission success Cognitive simulation Behavior modifiers Situation Awareness Error, Workload Timeliness Operator Characteristics Performance measures

39. Typical Outputs

40. “Fresh”

41. “Tired”

42. PC Version: Early Simulation

43. Conclusion • MIDAS 3.0 now operates on a PC platform and will soon incorporate significantly enhanced cognitive model (Apex) • MIDAS 3.0 gives users the ability to model the functional and physical aspects of a variety of operators, systems, and environments. • It brings these models together in an interactive, event-filled simulation for quantitative and visual analysis • The interplay between top-down and bottom-up processes and a suite of performance modifying functions enables the emergence of un-forseen, un-scripted behaviors • The government has done what it set out to do - - spur development of human performance modeling tools integrated into a design environment • Our goal is to continue to add functionality with each new application