Understanding Mental Workload Assessment Techniques

1. Measuring the Load of Mental Work Methods and Techniques P.A. Hancock Presentation for the Class of 2008 Human Factors II EXP 6257 January 24th, 2008 Department of Psychology s Institute for Simulation and Training University of Central Florida Orlando, FL 32826

2. Measuring the Load of Mental Work Structure of the Lecture: i) What is Mental Workload? ii) Why Measure Mental Workload? iii) Methods to Measure Mental Work. iv) Current Applications and Future Directions. Hancock, P.A., & Meshkati, N. (1988). (Eds.). Human mental workload. North-Holland: Amsterdam Available at: www.mit.ucf.edu

3. What Is Mental Workload? • Mental workload is the portion of operator’s limited mental capacities actually required to perform a particular task. • Mental reserves are the difference between capacity required and capacity available. • Mental effort is the voluntary matching of mental capacities with that needed for task success. • Increase in Mental Workload often precedes Performance Failure.

4. The Need To Measure Mental Workload Comes From The Changing Nature of Work

5. But Still Work Overload

6. How Hard Are These People Working? Apparent Underload Apparent Overload

7. Measuring the Load of Mental Work There are FOUR Basic Methodologies: i) Primary Task Performance. ii) Subjective Response. iii) Physiological Assessment (Central and Peripheral). iv) Secondary Task Techniques. Meshkati, N., Hancock, P.A., Rahimi, M., & Dawes, S.M. (1995). Techniques of mental workload assessment. In: J. Wilson and E.N. Corlett, (Eds.). Evaluation of human work: A practical ergonomics methodology. London: Taylor & Francis. Available at: www.mit.ucf.edu

8. Measuring the Load of Mental Work Primary Task Performance Hancock, P.A., & Meshkati, N. (1988). (Eds.). Human mental workload. North-Holland: Amsterdam Available at: www.mit.ucf.edu

9. Primary Task Techniques Measures the Performance Outcome as a function of Primary Task Demand How well are you Driving? How well are you Flying? Notice that its not so easy to specify ‘well’ in complicated performance environments.

10. Primary Task Techniques As task load increases, the additional demands on mental capacities result in a degradation in performance Advantages of This Measure: Workload reflected directly by performance outcome. Non-invasive and non-interfering. Tracks changes in workload dynamically. (i.e., as performance proceeds) Uncontaminated by memory issues Disadvantage of This Measure: Only sensitive to changes in workload at the limits of mental capacity If operators can compensate for increased workload by increasing their Effort, the primary task measure is insensitive Mental Workload not distinguished from performance outcome

11. Primary Task Techniques What About Failure? Most of the time we’d like to know about Mental Workload to know how much is too much? Primary Task measures do not tell us this. So, - they fail to be informative just at the time they are needed most!

12. Secondary Task Performance Multiple Task Demands

13. Secondary Task Measures Maximum Capacity (Without Impaired Performance). S1 S3 S2 S2 Performance Level S1 P S3 P E M D P PTD Easy (E) Moderate (M) Difficult (D) Primary Task Demand (PTD)

14. Measuring the Load of Mental Work Subjective Measures Hancock, P.A., & Caird, J.K. (1993). Experimental evaluation of a model of mental workload. Human Factors, 35, 413-429. Available at: www.mit.ucf.edu

15. Subjective Responses • If You Want to Know How Hard Someone is Working – Ask Them: • You Can Formally ‘Ask Them’ Through Standard Techniques. • Two of the Most Popular are SWAT and NASA-TLX. • The Advantage: It is Easy to Do and Has high Face Validity • The Disadvantage: Often Performance and Perception Deviate

16. Subjective Workload Assessment Technique (SWAT). • Developed by the United States Air Force. • There are THREE Sources of Workload: Time, Effort, and Stress. • Each has THREE Levels 1=Low, 2=Medium, 3=High. • You begin by Putting the 27 Cards • (3 Sources X 3 Levels X 3 Combinations) into Order from 1-27. • Subjects rate each EVENT by giving a number for each, • (e.g., Time=2, Effort=1, Stress=3). • Looking up this Combination in the Card Sort Gives you • the Workload on a 0-100 Scale.

17. NASA-TLX (Task Load Index). • Developed by NASA (duh!). • There are SIX Sources of Workload: • Temporal Demand, Effort, Stress, Own Performance, Frustration, Physical Demand • Each is Compared Pairwise against the Others to give a • Rank Order (0-5). • Subjects rate each EVENT by giving a 0-100 score for each • Source. • These values are multiplied by the RANK and the total • is divided by 15 to get the Workload Score on a 0-100 Scale.

18. Measuring the Load of Mental Work Physiological Measures Central vs. Peripheral NS Measures Hancock, P.A., Meshkati, N., & Robertson, M.M. (1985). Physiological reflections of mental workload. Aviation, Space, and Environmental Medicine, 56, 1110-1114. Available at: www.mit.ucf.edu

19. Physiological Reflections of Mental Workload Electroencephalography (EEG) Pupillometry/ Eyetracking Electrocardiography (ECG) Eyeblink & Reflex Modification

20. Pupil Diameter Kahneman & Beatty (1966; 1967); Hakerem & Sutton (1966); Hess & Polt (1964)

21. Visual Scanning and Workload • Used extensively in aviation research • Tole, et al. (1982) • Visual scanning behavior and mental workload in aircraft pilots • gaze characteristics on cockpit instruments varied as a function of the level of difficulty of a verbal loading task • mean dwell time of each fixation on the pilot's primary instrument increased as a function of task load • scanning behavior was also a function of the estimated skill level of the pilots, with novices being affected by the loading task much more than experts. • Authors argue that visual scanning of instruments in a controlled task may be an indicator of both workload and skill

22. Visual Scanning and Workload Head Mounted Version Desk Mounted Version

23. Time-Based ECG and Workload • Bonner & Wilson (2001) • Monitored pilots throughout test and evaluation of an aircraft • Note differences between subjective workload and HR

25. Measurement Techniques: Advantages and Disadvantages i) Primary Task Performance. (Data Easily Available,Future Failure Unpredictable). ii) Secondary Task Technique. (Diagnostic, Administration is Intrusive).. iii) Subjective Measures. (High Face Validity, Often Dissociate). iv) Physiological Assessment. (Unobtrusive, Expensive Data [but getting cheaper]).

26. Current Applications and Future Directions • Employee burnout • Training effectiveness • Human computer interaction • Home-care • Nursing • Hospital readmissions • Parenting • Consumerism • Professor productivity • to improve academic quality • Student success • Aviation • Pilot workload • Maritime • Ship navigation • Ground • Car and bus driver’s workload • Air traffic control • “Automation cueing modulates cerebral blood flow and vigilance in a simulated air traffic control task” • Shift-work • Performance dependent upon shift and workload • Peacekeeping • Differential workload of peacekeepers • Business costs • “… employee benefits managers are hoping that technology … will help them cope with increasing workloads” To name just a few...

27. CURRENT APPLICATIONS AND FUTURE DIRECTIONS Variance in Shear Forces After: Marras, W. (2005), Ohio State University Combined Physical and Mental Workload If 20% of the Forces calculated in Physical Work are Personality Factors, what is that number for Cognitive Work. After: Marras, W. (2005), Ohio State University

28. Neuroergonomics “Neuroergonomics involves the examination of the neural bases of perceptual, cognitive and motor functions in relation to real-world applications as mediated through Machines.” Adaptive Human-Machine Systems Hancock, P.A. (1997). On the future of work. Ergonomics in Design, 5(4), 25-29.

29. RESOLUTION SPACE OF BRAIN IMAGING TECHNIQUES FOR ERGONOMIC APPLICATIONS More invasive, Less practical TCDS 20 cm EEG, ERPs 10 cm NIRS SPATIALRESOLUTION PET Less invasive, More practical 1 cm MEG 5 mm fMRI 0.1 mm 1 ms 10 ms 2 s 2 min TEMPORAL RESOLUTION

30. Workload and Vigilance Vigilance is a Long-Standing Problem. Sources of Performance Influence Include: Event Rate Signal Salience Stress/Workload/Fatigue Glare, Noise, Temperature, Vibration, TOD, Drug Effects etc Memory Load Successive vs. Simultaneous Comparisons Feedback Hit vs. Miss vs. FA KR/KP Individual Differences Introversion/Extraversion, Age, Sex, Expertise Warm, J.S. (Ed.). (1984). Sustained attention in human performance. New York: Wiley.

31. P.A. Hancock, D.Sc., Ph.D. Department of Psychology, and The Institute for Simulation and Training University of Central Florida Orlando, FL 32826 407-823-1492 phancock@pegasus.cc.ucf.edu www.mit.ucf.edu

32. Orlando, Florida, January, 2007. Human Factors II Personal Biography Peter A. Hancock Professor of Psychology and Institute for Simulation and Training, University of Central Florida The Load of Mental Work Peter Hancock is Provost Distinguished Research Professor in the Department of Psychology, the Institute for Simulation and Training, and at the Department of Civil and Environmental Engineering at the University of Central Florida. He also holds courtesy appointments at Minnesota, Michigan, and MIT. He is the author of over five hundred refereed scientific articles and publications including: Human Performance and Ergonomics; Stress, Workload, and Fatigue; and Essays on the Future of Human-Machine Systems. He has been continuously funded by extramural sources for every year of his professional career, including support from NASA, NIH, NIA, FAA, FHWA, the US Navy, the US Army and the US Air Force. In 2000 he was awarded the Sir Frederic Bartlett Medal of the Ergonomics Society of Great Britain for lifetime achievement. He was the Keynote Speaker for the 2000 Meeting of the International Ergonomics Association. In 2001 he won the Franklin V. Taylor Award of the American Psychological Association and in association with his colleagues Raja Parasuraman and Anthony Masalonis, he was the winner of the Jerome Hirsch Ely Award of the Human Factors and Ergonomics Society for 2001. In 2002, he was awarded the Jastrzebowski Medal of the Polish Ergonomics Society for contributions to world ergonomics and in the same year was named a Fellow of the Ergonomics Society of Great Britain. He has been elected to a second, three-year term as a Member of the National Research Council’s Committee on Human Factors. In 2003 he won the Liberty Mutual Medal of the International Ergonomics Association. His current experimental work concerns the evaluation of time an behavioral responses to high-stress conditions. His theoretical works concerns human relations with technology and the possible futures of this symbiosis. He is a Fellow of and past President of the Human Factors and Ergonomics Society. Further Information can be garnered at: www.mit.ucf.edu

35. Workload-Performance Dissociations Measure of Subjective Workload Supply of Resources Underload Measure of Performance Overload Task Demand (difficulty)

36. With the Transition from Physical to Mental Work Comes the Need to Measure Workload All Citations Available At: www.mit.ucf.edu

37. Technology is a way of organizing the universe so that man doesn't have to experience it. -- Max Frisch Workload: Applied • Aviation - Automation & Workload in Aviation Systems • The superiority of Adaptive Automation • “The allocation of a task or a function between the operator and the system [which] is flexible and responsive to operator’s performance and level of workload” • “Improve situational awareness, regulate workload, improve vigilance in high-risk environments, and help to maintain manual control skills. (Mouloua, Deaton, & Hitt in Hancock & Desmond, 2001; Parasuramanm, Bahri, Deaton, Morrison, & Barnes, 1992) • Adaptive Automation • “Adaptive automation (AA) for managing operator workload through dynamic control allocations between the human and machine over time” • Low-levels of automation - superior performance • Intermediate levels of automation - higher SA • Not associated with • Improved performance • Reduced workload. • (Kaber & Endsley, 2004)

38. Human Performance under Stress • Information Overload/Complexity • Physical/Social Environment • Uncertainty

39. EEG