Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh

1. DEVELOPING A TRAINING PROGRAM FOR THE TRAFFIC ALERT AND COLLISION AVOIDANCE SYSTEM IN CONTEXT Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr. Ute Fischer Sponsored by the FAA, Tom McCloy as Technical Monitor

2. Overview Develop a training program intended to improve pilots’ understanding of TCAS use for collision avoidance in a range of traffic situations Introduction to TCAS and Background Information Design of Training Program Evaluation of the Training Program Results Conclusions

3. Traffic alert and Collision Avoidance System (TCAS) • Pilot always shown the Traffic Situation Display (TSD) • TCAS delivers a two stage advisory and vertical avoidance maneuver • Traffic Advisory (TA) - ‘Traffic Traffic’ • Resolution Advisory (RA) - ‘Climb Climb’ • Pilot is to follow an RA, even if it conflicts with ATC instructions, unless the pilot believes that safe flight would be jeopardized Federal Aviation Administration (2000). Introduction to TCAS II Version 7. Washington, D.C.

4. Advised Maneuvers for Collision Avoidance Notification and Awareness Pilot’s Response Collision Avoidance in a Broader Context TCAS Traffic Advisory Compliance to TCAS ATC Traffic Callout TCAS Resolution Advisory Compliance to ATC Visual acquisition of other aircraft ATC Maneuver Personal Assessment and Maneuver Decision Awareness of other aircraft via TSD Party-line Information Pilots don’t only interact with TCAS, they also receive information from ATC and the environment

5. Current TCAS Training Requirements: Ground Training To comply to the RA, you should Pull the stick back Push the stick forward Federal Aviation Administration (2001). Advisory Circular No. 120-55B: Air Carrier Operational Approval and Use of TCAS II. Washington, D.C. Classroom or computer based >60 training requirements Measures performance through quizzes and/or activities

6. Current TCAS Training Requirements: Flight Training

7. Observed TCAS Use and Operation Non-Compliance TOPA observed compliance rates of 41% to Climb RA’s Misunderstanding/ Confusion 59% to Descend RA’s Aggressiveness Olson, W. and J. Olszta (2010).TCAS Operational Performance Assessment in the U.S. National Airspace. IEEE/AIAA Digital Avionics Systems Conference. Olszta, J., & Olson, W. (2011). Characterization and Analysis of Traffic Alert and Collision Avoidance Resolution Advisories Resulting for 500' and 1,000' Vertical Separation. Paper presented at the Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011), Berlin, Germany. • The TCAS Operational Performance Assessment (TOPA) monitored the terminal area of 8 major airports and examined pilot compliance to climb and descend advisories

8. Observed TCAS Use and Operation Pilot reports flying into the red region on the VSI “[My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a person should fly 'away' from the red VSI indication, if instructed via RA.” ACN:785761, 2008 Non-Compliance Misunderstanding/ Confusion Pilot disagrees with advised descend RA Aggressiveness “Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the initial guidance was so clearly wrong” ACN: 854982, 2009 NASA. (2009). Aviation Safety Reporting System. Retrieved August 1, 2010: http://asrs.arc.nasa.gov/

9. Observed TCAS Use and Operation Far Eastern Air B757 Response to Descend RA (TCAS advised a descent rate of 1500 FPM) Non-Compliance Misunderstanding/ Confusion Aggressiveness Image copied from Lacagnina (2008). Easy Does It. Aero Safety World : http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1

10. Research Statement and Objectives (1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment (1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment (2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories (2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories Develop a training program intended to improve pilots’ understanding of TCAS use for collision avoidance in a range of traffic situations

11. Approach to Training Design Complement of Two Methods Demonstration Based Training Event Based Training

12. Demonstration Based Training (DBT) • Computer-Based Training • ~25 minutes • Provides conceptual understanding of TCAS • Outlines rules for compliance • 6 segments • Introduction to TCAS • Traffic Situation Display • TCAS Advising Logic • Traffic Advisories • Resolution Advisories • Example Timeline of RA Evolution • Mid-Training Quizzes

13. DBT: Demonstrations of RA’s

14. Event Based Training (EBT) • Presents traffic events that create the requirement to act • Builds context and complexity into each scenario as the flight progresses • Uses a more realistic training environment Fowlkes, J., Dwyer, D., Oser, R., & Salas, E. (1998). Event-Based Approach to Training (EBAT). The International Jounal of Aviation Psychology, 8(3), 209-221.

15. Simulator Study in Integrated Flightdeck – ATC Environment Audio Communications (Aviation Intercom) SideStick ATC->TSD VGA Simulation Architecture Air Traffic Simulator (TGF) PFD ATC Captain TSD Air Traffic Transcripts Charts & Checklists Eyetracker ND PartyLine B747-400 Simulator (RFS) Touch screen First Officer PFD TCAS Logic TCAS Alerts ND Coded Log of Flights Experimenter/Instructor Pritchett, A., Fleming, E., Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012). Pilot Interaction with TCAS and Air Traffic Control. Paper presented at the 2nd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS), London.

16. Structure of Events: Descend RA Example

17. Evaluating the training program’s impact

18. Overview of Study Comparison to identify the impact of training • Baseline Study • Training Study • 6 scenarios with 2 traffic events per scenario • Traffic events defined by • RA type • ATC information • Traffic density

19. Overview of Study Pre-Training Data Collection (50 minutes) Pre-Experiment Questionnaire Pre-Experiment Quiz Introduction to TCAS TCAS Training Program (50 minutes) Demonstration Based Training Short Section Quizzes Event Based Training Evaluating TCAS Training Program (120 minutes) Flight Scenarios Post Scenario Questionnaires Debrief (15 minutes) Post- Experiment Questionnaire

20. Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS? Skill-Based Behavior Skill-Based Behavior Rule-Based Behavior Rule-Based Behavior Knowledge-Based Behavior Knowledge-Based Behavior

21. Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS? • Decrease time to achieve compliance • Decrease time to disconnect autopilot • Reduce aggressive response features • Increase percentage of RA duration in compliance • Increase appropriate response post-Clear of Conflict • Increase understanding of TCAS • Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

22. Assessing the Pilot’s Response: Skill-Based Behaviors Pilot’s Response TCAS RA Maneuver Vertical Speed X 5 seconds time 2 ½ sec Clear of Conflict Time Pilot First Achieves Compliance TCAS assumed constant vertical RA rate Autopilot Disconnect Time TCAS weakens required vertical rate RA Climb TCAS assumed ¼ g pull-up Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

23. Mean Time to Comply: During Training pMM = Significance for the mixed model ps2 = Significance of the variance pm = Significance of the means Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

24. Autopilot Disconnect: Post Training Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior • During training, no significant differences • Post training • Autopilot disconnect time decreased • Pilot response was more consistent for one event

25. Impact of Training Program • Decrease time to achieve compliance • Decrease time to disconnect autopilot • Reduce aggressive response features • Increase percentage of RA duration in compliance • Increase appropriate response post-Clear of Conflict • Increase understanding of TCAS • Increase trust in TCAS Skill-Based Behavior Skill-Based Behavior Rule-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

26. Assessing the Pilot’s Response: Aggressiveness Pilot’s Response TCAS RA Maneuver Maximum vertical rate *Altitude Deviation Maximum vertical rate difference Vertical Speed Vertical rate difference 5 seconds time 2 ½ sec Clear of Conflict TCAS weakens required vertical rate RA Climb TCAS assumed ¼ g pull-up TCAS assumed constant vertical RA rate Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

27. Assessing the Pilot’s Response: Compliance Pilot’s Response TCAS RA Maneuver Not in Compliance In Compliance Vertical Speed 5 seconds time 2 ½ sec Clear of Conflict TCAS weakens required vertical rate RA Climb TCAS assumed ¼ g pull-up TCAS assumed constant vertical RA rate Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

28. Assessing the Pilot’s Response: Return to Clearance Pilot holds new altitude Contacts ATC and request new clearance or ask for further instructions Pilot’s Response TCAS RA Maneuver Altitude Pilot begins descent back to originally cleared altitude May inform ATC of response to RA Cleared altitude at time of RA time 5 seconds RA Climb TCAS assumed ¼ g pull-up TCAS assumed constant vertical RA rate TCAS weakens required vertical rate Clear of Conflict Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

29. Before Training “[Pilots] must always comply with a TCAS RA unless [aircraft] performance is hindered (i.e. operating single engine) or [there is] an obvious TCAS malfunction (ie you can see traffic and it is not a threat)” Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior • 89% of the pilots did not know that an RA should cause less than 500 feet of altitude deviation • 56% of the pilots responded that they would hold current altitude achieved after responding to an RA, as opposed to returning to their clearance • When asked about airline procedures for following TCAS, all 18 pilots noted the need for complying with an RA • BUT 28% of the pilots commented compliance wasn’t necessary is there was a TCAS “malfunction” or if the RA would cause an “unsafe situation”

30. Aggressiveness Altitude Deviation Average Vertical Rate Difference Maximum Vertical Rate Difference Maximum Vertical Rate Difference All decreased Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior During training, aggressive response features decreased Post training, same decreasing trend observed

31. Percentage Compliance Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior • During training, percentage compliance decreased in training event with “Climb RA” (93.1% compared to 99.6%) • Post training • No significant differences in means observed • But, trained pilots had a more consistent response

32. Return to Clearance Pilot returns to original clearance Yes No Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior

33. Impact of Training Program • Decrease time to achieve compliance • Decrease time to disconnect autopilot • Reduce aggressive response features • Increase percentage of RA duration in compliance • Increase appropriate response post-Clear of Conflict • Increase understanding of TCAS • Increase trust in TCAS Skill-Based Behavior Rule-Based Behavior Rule-Based Behavior Knowledge-Based Behavior Knowledge-Based Behavior

34. Before Training Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior • 94% pilots agreed with statement “I understand TCAS maneuvers when they are issued”, but… • Only 50% of the pilots correctly identified the assumptions made by TCAS advisory logic • When asked to interpret TSD symbols, only 50% of the pilots got all parts of the associated questions correct

35. Post Training Understanding of TCAS 61% 33% 6% 0 0 Strongly Disagreed Disagreed Neutral Agreed Strongly Agreed • 50% pilots reported an increase in understanding TCAS logic • 27% pilots claimed to have learned about different types of TCAS RA’s (notably, “Crossing RA’s”) “Types of RA's were not previously taught. We were taught simply to comply” Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior “My understanding of TCAS has increased”

36. Post Training Trust in TCAS 39% 39% 11% 11% 0 Strongly Disagreed Disagreed Neutral Agreed Strongly Agreed “My trust in TCAS was already at a maximum so I wouldn’t be ‘more’ likely to trust it” Skill-Based Behavior Rule-Based Behavior Knowledge-Based Behavior “I am more likely to trust TCAS after completing today’s training”

37. Post-Training • Decrease time to achieve compliance • Decrease time to disconnect autopilot • Reduce aggressive response features • Increase percentage of RA duration in compliance • Increase appropriate response post-Clear of Conflict • Increase understanding of TCAS • Increase trust in TCAS

38. Conclusions and Future Work Conclusions • Pilots may not need more training, but instead need better training • Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive • EBT structure allows for the design of purposeful training events • Current FAA mandated TCAS training objectives may not fully reflect all training areas • Language needed to address the reduction of excessive responses to RA’s Future Work • What facilities and technologies would be required to implement this type of training program? • What implications arise when considering training design versus system design? • Would incorporating human factors considerations in initial design stages decrease the amount of required training?

40. Conclusions and Future Work Conclusions • Pilots may not need more training, but instead need better training • Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive • EBT structure allows for the design of purposeful training events • Current FAA mandated TCAS training objectives may not fully reflect all training areas • Language needed to address the reduction of excessive responses to RA’s Future Work • What facilities and technologies would be required to implement this type of training program? • What implications arise when considering training design versus system design? • Would incorporating human factors considerations in initial design stages decrease the amount of required training?

41. Acknowledgements Work sponsored by the FAA, Tom McCloy as Technical Monitor 34 pilot participants Dr. Amy Pritchett Dr. Karen Feigh Dr. Ute Fischer Dr. Wesley Olson, MIT Lincoln Labs Wayne Gallo, FAA Roger Sultan, FAA Kylie Garey TCAS Team: William Cleveland, VladPopescu, Justin Mullins, Anil Bozan, Henry Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, DhruvThaakar, Jonathan Zoetrum, JelleWissink CEC Lab Members