In Search of the FLA: LOSA, DOSA, ASAP

1. In Search of the FLA:LOSA, DOSA, & ASAP Human Factors Research Project The University of Texas at Austin February 2003

2. LOSAJames Klinect

3. Threat and Error Management LOSA (1997-2002) Continental Latin America Continental Express Gulfstream Express Air New Zealand Air Micronesia Continental Delta USAirways Cathay Pacific EVA Air Uni Air (Taiwan) Frontier QANTAS Braathens Forthcoming Alaska Singapore

4. LOSA Components Part 1. Flight crew survey Part 2. Flight crew interview Part 3. Descent and approach Part 4. CRM Countermeasures Behavioral markers Part 5. Threat and error mgt

5. Representative Threat Findings 90% of flight segments had one or more threats Organizational range � 84% to 97% Most frequent threats Adverse weather � 19% ATC clearances/late changes 16%

6. Error and Error Response 63% of flights observed had one or more errors 45% Undetected 29% Detected and action taken 5% Detected and ignored

7. Error Outcomes Inconsequential � 71% Consequential � 29% Additional Errors � 8% (Error Chains) Undesired Aircraft States � 21%

8. Phase of Flight Effects Descent / approach / land phase contains the most variability in crew performance and consequential errors

9. Organizational Differences

10. Threat and Error Relationship between number of threats and errors is relatively weak, although statistically significant (r=.19)

11. A model of Threat and Error Management derived from LOSA data The model is being used as a the framework for the analysis of ASAP reports, incidents, and accidents

12. Threat and Error Management Model

13. Safety Culture Scale Pilots observed in LOSA also completed a survey with items from the UT Flight Management Attitudes Questionnaire Items related to organizational culture regarding safety were summed to form Safety Culture Scale

14. Safety Culture Scale Items: Read the itemsRead the items

15. Recognition and Extension of LOSA ICAO LOSA HANDBOOK issued June 2002 LOSA to become ICAO standard by 2005 Will the FAA comply with ICAO requirement? The UT Threat and Error Management Model adopted as framework for analysis of global transport accidents by IATA, 2003 LOSA presented to North American Safety Conference on panel with FAA Administrator Blakey, February 2003 Dispatch LOSA (DOSA with Continental)

16. DOSA:Dispatch Operations Safety AuditChris Henry

17. Dispatch Safety Several airlines have implemented ASAP programs for their dispatchers. Continental Airlines approached UT about creating a LOSA-like approach to gathering safety data in dispatch operations. The DOSA methodology aims to collect data on safety (threats & errors) and operational matters (fuel & on time performance)

18. Dispatch Threats Departure/Arrival: WX, terrain, congestion, airport conditions Aircraft Event: MEL�s, aircraft malfunction, communication event Operational: time pressures, diversion, unfamiliar airport (e.g., charter flights) Cabin: cabin event (MEDLINK) ATC: Routing changes Crew support: MX, ground, dispatch, scheduling, load planning. Listing of threats that the crew may encounter in flight.Listing of threats that the crew may encounter in flight.

19. Dispatchers���� Must deal with many of the threats that pilots must deal with. Encounter other threats are specific to the dispatcher and are not necessarily encountered by the crew. Are responsible for planning and following between 30-50 flights per shift. Work in a threat rich environment and can be thought of as threat managers. Particularly if you think of flight following. The D is to keep the crew apprised of changing conditions so that they will have the most current information in the most timely manner in able to make prudent decisions. In what situations are the crew likely to contact their dispatcher? It�s not going to be to say �hello.� It�s because they have some sort of question or problem that the D can help with. Particularly if you think of flight following. The D is to keep the crew apprised of changing conditions so that they will have the most current information in the most timely manner in able to make prudent decisions. In what situations are the crew likely to contact their dispatcher? It�s not going to be to say �hello.� It�s because they have some sort of question or problem that the D can help with.

20. Two Levels of Threat The flight: Threats that are specific to one flight The desk: Threats that are not tied to one specific flight but that could potentially affect all of the flights on at the dispatcher�s workstation (desk)

21. Errors Nearly 80% of errors in cockpit LOSA�s relate to selecting and following the right procedures Dispatch is much less proceduralized than flightdeck operations Processes may be more telling than errors. In other words, what is lost first when conditions go south. Dispatch is, by design, less proceduralized that flightdeck operations. This is due to the extremely dynamic nature of dispatch operations and the number of flights on a desk. If strict procedures do not exist, it is difficult to say that someone has committed an error (In some cases, such as applying MEL�s and NOTAM�s, this can be done). Yet from my observations, matters are tended to with differing levels of quality. This quality is partially dependent on the threats that are present. Processes may be more telling than errors. In other words, what is lost first when conditions go south. Dispatch is, by design, less proceduralized that flightdeck operations. This is due to the extremely dynamic nature of dispatch operations and the number of flights on a desk. If strict procedures do not exist, it is difficult to say that someone has committed an error (In some cases, such as applying MEL�s and NOTAM�s, this can be done). Yet from my observations, matters are tended to with differing levels of quality. This quality is partially dependent on the threats that are present.

22. A New Direction�. Process How does the dispatcher deal with threats in good conditions? What is lost when conditions get bad? Which methods are associated with saving the company money by more efficient fuel use and on time performance?

23. Data Collected Pilot data: 33 observations conducted last fall Baseline data: 23 observations collected during pristine weather conditions in early January Future data collection: aim to collect data during thunderstorm season Survey data: 58 dispatchers returned survey designed to measure attitudes pertaining to safety

24. Roundtables Examine data on all threats and errors to validate their legitimacy Conducted at UT with Continental Dispatch Management Representative Continental Dispatch Union Representative Continental Safety Manager University of Texas personnel

25. ASAPMichelle Harper

26. Aviation Safety Efforts Non jeopardy reporting Proactive safety programs LOSA (Flight Operations, Dispatch) and ASAP Goals of Proactive, Non-Jeopardy Safety Programs 1. Collect voluntary safety information that may be used to identifying potential precursors of accidents 2. Use this information to develop corrective actions to reduce the potential for accidents, incidents and other safety-related problems Data Safety Change Process We are currently developing research projects that involve non-jeopardy reporting Any type data collection (either automated or self-report) that is used to identify high risk safety issues without risk to the reporter or any individuals involved. - Use information known by pilots to improve the system without putting them at risk for punitive action Last 8 to 10 year there has been a large increase in non-jeopardy reporting to create proactive approach to safety Two systems we will be talking about today �our Line Operation Safety Audit program (both Flight Operations and Dispatch) and our involvement with Aviation Safety Action Program What are the goals of these research projects: To collect information that would not be known without the consent of pilot To use information to identify hazardous issues before they become critical or lead to an accident Our goal has been to develop a working model that will enable airlines to complete these two goals What we have seen is that there is a huge push to collect as much information as possible, airlines have stock piles of information about what is going on How do you move from Goal 1 to Goal 2: Problem how do you take these piles of information from thousands of different flights, pilots, operational environments and develop a set of workable corrective actions that can be used to feed the safety change process. We are currently developing research projects that involve non-jeopardy reporting Any type data collection (either automated or self-report) that is used to identify high risk safety issues without risk to the reporter or any individuals involved. - Use information known by pilots to improve the system without putting them at risk for punitive action Last 8 to 10 year there has been a large increase in non-jeopardy reporting to create proactive approach to safety Two systems we will be talking about today �our Line Operation Safety Audit program (both Flight Operations and Dispatch) and our involvement with Aviation Safety Action Program What are the goals of these research projects: To collect information that would not be known without the consent of pilot To use information to identify hazardous issues before they become critical or lead to an accident Our goal has been to develop a working model that will enable airlines to complete these two goals What we have seen is that there is a huge push to collect as much information as possible, airlines have stock piles of information about what is going on How do you move from Goal 1 to Goal 2: Problem how do you take these piles of information from thousands of different flights, pilots, operational environments and develop a set of workable corrective actions that can be used to feed the safety change process.

27. Aviation Safety Action Program Currently 30 programs/18 airlines who have submitted MOUs to the FAA Includes Flight Ops, Dispatch and Maintenance Goals of ASAP 1. Collect voluntary safety information that may be used to identify potential precursors of accidents. 2. Use this information to develop corrective actions to reduce the potential for reoccurrence of accidents, incidents and other safety-related problems. Joint agreement between FAA, ALPA, and safety manager from within airlines Identified non-jeopardy pilot reporting system that requires group of FAA, ALPA and carrier to review each report and make corrective action recommendations We have seen a huge increase in the number of programs who have submitted MOUs Size has doubled in the last year Main goals of ASAP same as those of any non-jeopardy reporting system Types of data collection primarily narrative, similar to the ASRS report. American is doing a bit more looking at categorizing pilot error. Talk about what we have been doing with this program (specifically with CO in the last 18 months) Joint agreement between FAA, ALPA, and safety manager from within airlines Identified non-jeopardy pilot reporting system that requires group of FAA, ALPA and carrier to review each report and make corrective action recommendations We have seen a huge increase in the number of programs who have submitted MOUs Size has doubled in the last year Main goals of ASAP same as those of any non-jeopardy reporting system Types of data collection primarily narrative, similar to the ASRS report. American is doing a bit more looking at categorizing pilot error. Talk about what we have been doing with this program (specifically with CO in the last 18 months)

28. Threat and Error Management Model Application to ASAP The primary factors assessed: Type of event Pilot errors Operational threats Pilot performance markers Framework to collect, code, and retrieve relevant data Framework for analysis of underlying human and system deficiencies that contribute to an event Ok lets talk about the threat and error management model Two things: Theoretical model for assessing the relationship between primary factors that we see affecting aviation safety as it occurs during normal line operations. Primary factors we assess Type of event Error Threat Pilot Performance Markers We are interested in describing the environment as well as how the pilots manage the threats and errors they encounter We are interested in both the formal and informal process that crews use to manage various types of distractions from external sources as well as the errors that they themselves make Not a model based purely on assessing errors but that focuses what contributes to mistakes and how pilots manage unpredicted, non standard situations With the two goals of non-jeopardy reporting in mind we have developed: A Working model that can be used to assess, identify and analyze key factors that cause events and how crews manage or mismanage these events. Ok lets talk about the threat and error management model Two things: Theoretical model for assessing the relationship between primary factors that we see affecting aviation safety as it occurs during normal line operations. Primary factors we assess Type of event Error Threat Pilot Performance Markers We are interested in describing the environment as well as how the pilots manage the threats and errors they encounter We are interested in both the formal and informal process that crews use to manage various types of distractions from external sources as well as the errors that they themselves make Not a model based purely on assessing errors but that focuses what contributes to mistakes and how pilots manage unpredicted, non standard situations With the two goals of non-jeopardy reporting in mind we have developed: A Working model that can be used to assess, identify and analyze key factors that cause events and how crews manage or mismanage these events.

29. Threat and Error Taxonomies Why bother coding if you collect narrative description from pilot. Taxonomies enable systematic categorization of data Taxonomy enables long term stability of review process Enables identification of high risk events Enables identification of contributing factors � why Enables link to safety change process Threat and error taxonomies derived empirically from LOSA data are being used both for ASAP and LOSA (K-threat and K-error) Their power comes from the fact that they reflect what really happens on the line under non-jeopardy conditions So as I mentioned the majority of ASAP programs are using a basic ASRS form which assesses Type categorization and pilot narrative Why bother to use a model or go through the trouble of categorizing a report or set of information? Wouldn�t it be easier to just collect a written report of what is happened. Simply labeling the type of event does not support a proactive approach to safety In terms of the two goals labeling type simply says that you know where the problems are Tells you nothing about why or how to address so we are still at Goal #1 of collecting information Knowing independently what errors, threats, and type of event does not tell us why or enable us to focus on most critical issues Key to the use of the threat and error model is that it can be used to look at trends You can ask not only what occurred by why So as I mentioned the majority of ASAP programs are using a basic ASRS form which assesses Type categorization and pilot narrative Why bother to use a model or go through the trouble of categorizing a report or set of information? Wouldn�t it be easier to just collect a written report of what is happened. Simply labeling the type of event does not support a proactive approach to safety In terms of the two goals labeling type simply says that you know where the problems are Tells you nothing about why or how to address so we are still at Goal #1 of collecting information Knowing independently what errors, threats, and type of event does not tell us why or enable us to focus on most critical issues Key to the use of the threat and error model is that it can be used to look at trends You can ask not only what occurred by why

30. What can be done with the data- Trend analysis What can be done with the data- Trend analysis

31. ASAP Data Flow Chart ASAP tools Online pilot reporting form based on the Threat and Error Management Taxonomies ERC Incident review form ASAP manager form Key to the system we have developed is it allows an airline to focus their limited resources on fighting an intelligent battle Reality is change is very difficult but what we suggest is that it is impossible without Asking the right questions Systematically storing the data - reality is you cannot push every report through the safety change process Analyzing for trends Threat and Error Model will not tell you how to ACT but it will tell you where to focus your limited resources ASAP tools Online pilot reporting form based on the Threat and Error Management Taxonomies ERC Incident review form ASAP manager form Key to the system we have developed is it allows an airline to focus their limited resources on fighting an intelligent battle Reality is change is very difficult but what we suggest is that it is impossible without Asking the right questions Systematically storing the data - reality is you cannot push every report through the safety change process Analyzing for trends Threat and Error Model will not tell you how to ACT but it will tell you where to focus your limited resources

32. The ASAP Database Feb-Mar 2002 N= 106

33. LOSA, ASAP, and CRM

34. LOSA , ASAP and 6th Generation CRM LOSA and ASAP validate the role CRM plays in safety Error Avoidance Threat Management Error Management Undesired Aircraft State Management Four groups of countermeasures (CRM skills) show up in LOSA reflecting both superior and below standard crew performance Team building Planning Execution Review / Modify

35. Interpreting Error Data for CRM Procedural errors - may indicate poor workload management and/or poor procedures Communications errors - may reflect inadequate CRM (monitoring and challenging) or complacency Proficiency errors - suggest pressures to train and/or need for higher standards and better monitoring by check airmen Decision errors - may indicate need for more CRM training on expert decision making and risk assessment Violations - suggest poor procedures, weak captain leadership and/or a culture of non-compliance

36. Air Transport AssociationSupport ATA Human Factors Committee will meet at the University of Texas 2/28 Formal presentations by: UT Human Factors Research Project University of Central Florida

37. The University of Texas Human Factors Research Project www.psy.utexas.edu/HumanFactors