Real-Time Human Performance Monitoring for Air Traffic Control Safety

1. Automated Human Performance Monitoring for Air Traffic Control Safety through Bayesian Network Modeling and Video Surveillance Zhe Sun1, Jiawei Chen1, Rui Li1, Shuai Li2, Pingbo Tang1,*, Yongming Liu3 1School of Sustainable Engineering and the Built Environment, Arizona State University, USA 2School of Computing, Informatics, and Decision Systems Engineering, Arizona State University, USA 3School for Engineering of Matter, Transport & Energy, Arizona State University, USA

2. Outline Problem Statement Motivation Potential Solutions Conclusion 36th International System Safety Conference, Aug. 13 – 17, 2018

3. Problem Statement The national airspace and the airport become more crowded due to the increasing air traffic volume, the number of accidents/incidents increased significantly. Human error is considered as one of the major factors that contribute to more than 70% of all aviation accidents in the United States. 36th International System Safety Conference, Aug. 13 – 17, 2018

4. Problem Statement 36th International System Safety Conference, Aug. 13 – 17, 2018

5. Problem Statement Air Traffic Controller (ATC) Errors: Greater than 60% anomalous human behavior (i.e. distraction, fatigue) 36th International System Safety Conference, Aug. 13 – 17, 2018

6. Motivation How to quantitatively assess the probabilistic dependence between anomalous human behaviors and human errors that eventually lead to near-ground accidents/incidents is important for real-time risk prediction and assessment. Near-ground Accidents/ Incidents Anomalous Behaviors Human Errors 36th International System Safety Conference, Aug. 13 – 17, 2018

7. Motivation • The overall goal of this study could help answering two questions: • How human error occurs? • What type of human errors are more likely to occur due to anomalous behaviors? • How accident occurs? • What type of accidents/incidents are more likely to occur due to certain human errors? 36th International System Safety Conference, Aug. 13 – 17, 2018

8. Potential Solution Accident/Incident Report Analysis Video Surveillance 36th International System Safety Conference, Aug. 13 – 17, 2018

9. Potential Solution Video Surveillance – Distraction Detection Algorithm 36th International System Safety Conference, Aug. 13 – 17, 2018

10. Potential Solution Video Surveillance – Fatigue Detection Algorithm 36th International System Safety Conference, Aug. 13 – 17, 2018

11. Potential Solution Bayesian Network (BN) Modeling 36th International System Safety Conference, Aug. 13 – 17, 2018

12. Conclusion Table 1— Distraction and Fatigue Detection Results 36th International System Safety Conference, Aug. 13 – 17, 2018

13. Conclusion Table 2— Conditional Probability of Human Errors on Distraction and Fatigue Table 3—Conditional Probability of Most Frequent Accidents/Incidents on Human Errors 36th International System Safety Conference, Aug. 13 – 17, 2018

14. Conclusion • This research proposed a real-time fatigue and distraction monitoring of ATCs to ensure aviation safety through real-time video surveillance and BN modeling. • The developed BN also help quantify the correlation between anomalous human behaviors (distraction and fatigue), human errors and accidents/incidents. • The real-time human behavior monitoring algorithm uses facial expression detection techniques to capture the fatigue and fatigue of ATCs and use as input to feed into the developed BN to get real-time risk assessment during air traffic control. 36th International System Safety Conference, Aug. 13 – 17, 2018

15. Acknowledgment The research reported in this paper was supported by funds from NASA University Leadership Initiative program (Contract No. NNX17AJ86A, Project Officer: Dr. Kai Goebel, Principal Investigator: Dr. Yongming Liu). The support is gratefully acknowledged. The authors would also like to express our very great appreciation to Dr. Yezhou Yang for his valuable and constructive suggestions during the planning and development of this research work. His willingness to give his time so generously has been very much appreciated. 36th International System Safety Conference, Aug. 13 – 17, 2018

16. References Ameen, S. (2014). Review of Fatigue Systems and Implementation of Face Components Segregation, 5–8. FAA. (2005). Runway Incursion Trends and Initiatives at Towered Airports in the United States, FY 2001 through FY 2004, (August). Isaac, A., Shorrock, S. T., & Kirwan, B. (2002). Human error in European air traffic management: The HERA project. Reliability Engineering and System Safety, 75(2), 257–272. https://doi.org/10.1016/S0951-8320(01)00099-0 Kulkarni, S. S., Reddy, N. P., & Hariharan, S. I. (2009). Facial expression (mood) recognition from facial images using committee neural networks. BioMedical Engineering Online, 8, 1–12. https://doi.org/10.1186/1475-925X-8-16 Loft, S., Sanderson, P. M., Neal, A., & Mooij, M. (2007). Modeling and Predicting Mental Workload in En Route Air Traffic Control : Critical Review and Broader Implications. Human Factors, 49(3), 376–399. https://doi.org/10.1518/001872007X197017. O’Toole, A. J., Roark, D. A., Jiang, F., & Abdi, H. (2005). Predicting Human Performance for Face Recognition. Face Processing: Advanced Modeling and Methods, (2006), 293–319. Palermo, R., Jeffery, L., Lewandowsky, J., Fiorentini, C., Irons, J. L., Dawel, A., … Rhodes, G. (2017). Adaptive Face Coding Contributes to Individual Differences in Facial Expression Recognition Independently of Affective Factors. Journal of Experimental Psychology: Human Perception and Performance, 44(4), 503–517. https://doi.org/10.1037/xhp0000463 Pape, A. M., Wiegmann, D. A., & Shappell, S. (2001). Air traffic control (ATC) related accidents and incidents: A human factors analysis, 1–4. Shafique, Y., & Author, C. (2014). Managing the Performance of Air Traffic Controllers : Developing and Proposing a Conceptual Perspective, 6(7), 267–277.

17. Thank you Questions ? Contact Information Zhe Sun: zsun43@asu.edu Jiawei Chen: jchen311@asu.edu Rui Li: ruili11@asu.edu Pingbo Tang: tangpingbo@asu.edu Shuai Li: shuaili4@asu.edu Yongming Liu: Yongming.Liu@asu.edu