Aviation Safety Information Analysis and Sharing (ASIAS) for General Aviation (GA) ASIAS-GA FAA Technical Monitor – Mic

1. Aviation Safety Information Analysis and Sharing (ASIAS) for General Aviation (GA) ASIAS-GA FAA Technical Monitor – Michael Vu PI – Alan Stolzer, Ph.D. General Aviation Pilot Study (Task Lead - Tony Pickering, Ph.D.) Flight Risk Analysis Tool (Task Lead - Jim Cannon)

2. Today’s (E)GAPS Agenda • Discuss GAPS study (2011 version) results • Describe how E-GAPS will enhance results

3. Scenario This a/c does a takeoff and landing every 3 hours This a/c does 3 takeoffs and landings every hour IF, the conditions in PHAK graphic applied to both aircraft … The a/c rate reported per hour for the J-3 might be 9x higher than the Gulfstream, but is that an accurate measure of their relative safety?

4. GAPS Background & Objectives • The Missing & Wrong Denominator Problems • Hinkelbein, Neuhaus, Schwalbe, Dambier (2010). Letter to Editor. Aviation, Space and Environmental Medicine. • Nall Report (2009) • Aviation Safety Data Accessibility Study Index: A Report On Issues Related to Publish Interest in Aviation Safety Data. (1997).

5. GAPS Background & Objectives • The Missing & Wrong Denominator Problems • Hinkelbein, Neuhaus, Schwalbe, Dambier (2010). Letter to Editor. Aviation, Space and Environmental Medicine. • Nall Report (2009) • Aviation Safety Data Accessibility Study Index: A Report On Issues Related to Publish Interest in Aviation Safety Data. (1997).

6. GAPS Background & Objectives • The Missing & Wrong Denominator Problems • Hinkelbein, Neuhaus, Schwalbe, Dambier (2010). Letter to Editor. Aviation, Space and Environmental Medicine. • Nall Report (2009) • Aviation Safety Data Accessibility Study Index: A Report On Issues Related to Publish Interest in Aviation Safety Data. (1997).

7. Problem Detailed per flight GA activity data do not exist Solution Conduct a survey to gain representative sample data; infer to population

8. Purpose • Purpose of research was to study feasibility of developing methods and metrics to more appropriately assess GA safety by: • Obtaining/estimating GA flight activity • Comparing flight hour with per flight safety metrics • Secondary aim to investigate cost efficiency of modifying survey responder burden (i.e., survey length & modality)

9. Survey Design • First mailing (n=2000) evaluated the response rate and associated costs of 8 conditions - 4 survey versions (i.e., 1 full and 3 short) X 2 response modalities (i.e., web-based vs. choice of hard-copy or web-based) • Based on those results, second (n=2000) and third (n=6000) mailings were web only

10. Data Collection Process

11. Data Collection Process http://www.gapssurvey.com/

12. GAPS Safety Metric Data & Scrub

13. GAPS Descriptive Statistics

14. Metric Estimation Process Scrubbed GAPS Compute number of respondents for each type of 2010 flight activity question. Obtain total number of pilots from FAA database. Compute number of respondents that flew each type of flight activity in 2010. Obtain number of accidents for each type of flight activity from NTSB database. Calculate percentage of respondents (pilots) that flew each type of flight activity in 2010. Calculate per hour and per flight accident rate metrics for each type of flight activity. Compute typical (mean/median) 2010 hours/flights for each type of flight activity. Compare per hour and per flight accident rate metrics. Safety Metrics

15. Metric Estimation Results Per hour day/night accident ratio = 2.395 Per flight day/night accident ratio = 2.156 Example: Both diagrams suggest pilots flying during the day are involved in more accidents than pilots flying at night. The magnitude of the different accident rate between day and night flight appears less when the per-flight metric is used, as compared to the per-hour metric.

16. Metric Estimation Results Per hour VFR/IFR accident ratio = 5.751 Per flight VFR/IFR accident ratio = 1.719 Example: Both diagrams suggest pilots flying VFR are more likely to be involved in accidents than pilots flying according to an IFR flight plan. The magnitude of the different accident rate between VFR and IFR flight appears considerably less when the per-flight metric is used, as compared to the per-hour metric.

17. Metric Estimation Results Per hour VMC/IMC accident ratio = 1.156 Per flight VMC/IMC accident ratio = 0.574 Example: The per-hour diagram suggests pilots flying in VMC are slightly more likely to be involved in accidents than pilots flying in IMC; however, the per-flight diagram suggests that pilots flying in VMC areconsiderably less likely to be involved in an accident than those flying in IMC. The accident rate between VMC and IMC flight appears different in both magnitude and direction when the per-flight metric is used, as compared to the per-hour metric.

18. Metric Estimation Results Per hour <200HP/>200HP accident ratio = 1.529 Per flight <200HP/>200HP accident ratio = 0.903 Example: The per-hour diagram suggests pilots flying with less than 200HP are more likely to be involved in accidents than pilots flying with high HP; however, the per-flight diagram suggests that pilots flying with less than 200HP are slightly less likely to be involved in an accident than those flying with high HP. The accident rate between pilots flying aircraft with less than 200HP and those flying with high HP appears different in both magnitude and direction when the per-flight metric is used, as compared to the per-hour metric.

19. Metric Estimation Results Per hour experimental/non-experimental accident ratio = 2.686 Per flight hour experimental/non-experimental accident ratio = 2.415 Example: Both diagrams suggest pilots flying experimental aircraft are more likely to be involved in accidents than pilots flying non-experimental aircraft. The magnitude of the different accident rate between experimental and non-experimental appears lesswhen the per-flight metric is used, as compared to the per-hour metric.

20. Metric Estimation Results Per hour conventional gear/other gear accident ratio = 3.931 Per flight conventional gear/other gear accident ratio = 3.175 Example: Both diagrams suggest pilots flying conventional gear aircraft are morelikelyto be involved in accidents than pilots flying non-experimental aircraft. The magnitude of the different accident rate between flying conventional gear and flying other gear appears less when the per-flight metric is used, as compared to the per-hour metric.

21. Metric Estimation Results Per hour day/night accident ratio = 2.395 Per flight day/night accident ratio = 2.156 Per hour day/night accident ratio =1.711 Per flight day/night accident ratio =2.470 Mean-based calculations Median-based calculations

22. Metric Estimation Results Per hour VFR/IFR accident ratio = 5.751 Per flight VFR/IFR accident ratio = 1.719 Per hour VFR/IFR accident ratio =3.318 Per flight VFR/IFR accident ratio =1.276 Mean-based calculations Median-based calculations

23. Metric Estimation Results Per hour VMC/IMC accident ratio = 1.156 Per flight VMC/IMC accident ratio = 0.574 Per hour VMC/IMC accident ratio =0.729 Per flight VMC/IMC accident ratio =0.663 Mean-based calculations Median-based calculations

24. Metric Estimation Results Per hour <200HP/>200HP accident ratio = 1.529 Per flight <200HP/>200HP accident ratio = 0.903 Per hour <200HP/>200HP accident ratio =1.302 Per flight <200HP/>200HP accident ratio =0.812 Mean-based calculations Median-based calculations

25. Metric Estimation Results Per hour experimental/non-experimental accident ratio = 2.686 Per flight hour experimental/non-experimental accident ratio = 2.415 Per hour experimental/non-experimental accident ratio =2.494 Per flight hour experimental/non-experimental accident ratio =1.732 Mean-based calculations Median-based calculations

26. Metric Estimation Results Per hour conventional gear/other gear accident ratio = 3.931 Per flight conventional gear/other gear accident ratio = 3.175 Per hour conventional gear/other gear accident ratio =3.992 Per flight conventional gear/other gear accident ratio =2.984 Mean-based calculations Median-based calculations

27. Conclusions/Suggestions • Cold call surveys using postal mailing and electronic response modality is a feasible method to capture typical pilot flight activity • Web-based survey method is preferred – slightly higher response rate and much less cost • Choice of activity metric (hours vs. flights) does influence accident metric estimate computations!

28. Conclusions/Suggestions • Choice of “central tendency statistic” also can impact metric calculations • For flight activities in this study, a per-flight metric tended to reduce accident metric differences between flight conditions • Policies and funding for safety improvements should use the most appropriate metric calculations

29. 5 Specific E-GAPS Improvements Precision: Breadth: Precision and Breadth:

30. FRAT Flight Risk Analysis Tool

31. Purpose To develop a web-app that could be utilized within the GA community on cell phone devices as a tool for pilots to assess their overall risk factors involved with a proposed flight, AND to capture de-identified hazard data in order to enhance GA safety.

32. Prior Phase • Designed and implemented a standardized risk assessment tool based upon pre-departure hazard identification criteria • Rolled out first in the business aviation community. Entered partnership with NBAA. Tool is linked from their website.

33. FRAT Timeline

34. Tool Development • Review of 26 existing FRAT designs – captured best practices • Compact Format • User Friendly, Quick, Easily Understood • Standard Elements • Goal – Capture Data to Identify Common Risk • Team Oriented Approach – Include Operations, Flight & Maintenance • Based on SMS principles • RESULT: FRAT tool with risk assessment items

35. Flight Risk Assessment Tool Operations Oriented Risk Analysis Focused on Operational Risk Factors • Airport Environment • Flight Crew Experience • The Aircraft’s Performance and Maintenance • The Operating Environment

36. Flight Risk Assessment Tool Airport Environment CriteriaValue Range • Weather 0 – 4 • Runway 0 – 4 • Terrain 0 – 5 • Day/Night 0 – 3 • Crew Knowledge 0 – 2 • Level of Control (Tower) 0 – 2 • Approach 0 - 4

37. Flight Risk Assessment Tool Flight Crew Experience CriteriaValue Range • Aircraft Currency 0 - 4 • Duty Day* 0 – 4 • Prior Rest 0 – 4 • Flight Leg # 0 – 5 • Time Zone Crossing 0 – 5 • Stress Factor 0 - 5 * Includes Circadian Rhythm Factors

38. Flight Risk Assessment Tool The Aircraft’s Performance and Maintenance CriteriaValue Range • Performance Limited 0 – 3 • Recent Maintenance – MEL 0 - 3

39. Flight Risk Assessment Tool The Operating Environment CriteriaValue Range • International 0 – 3 • Enroute Weather 0 – 3 • Non-Radar 0 – 3

40. NBAA Unique Perspective • NBAA Represents > 6,000 Flight Operations • 75 % of NBAA Members are Single Aircraft Operations • Majority of Which Fly Under FAR Part 91 Rules • NBAA Website Provides a Unique Forum for FRAT • NBAA Safety Committee Unanimously Endorsed FRAT • NBAA Members Have Open Access Through Website • Lessons Learned Are Easily Related to GA Community

41. FRAT Tool • Web and Mobile Optimized • Zero Learning Curve • Anonymous Trust

42. DEMO http://www.aircraftmerchants.com/frat http://aircraftmerchants.com/frat/firststats.cfm

43. Current Steps • Move host to cloud • Conduct several case studies • Develop and modify tool as appropriate for other sectors / groups • University flight programs • Aircraft type groups • AOPA • …. • Roll out to other sectors of GA • Collect and mine data • Develop public reporting site

44. Public Reporting

45. Public Reporting As of June 3, 2012