1 / 56

William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009

COMPARING THE ACCURACY OF PERFORMING DIGITAL AND PAPER CHECKLISTS USING A FEEDBACK INTERVENTION PACKAGE DURING NORMAL WORKLOAD CONDITIONS IN SIMULATED FLIGHT. William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009. Overview. Rationale & Purpose

iorwen
Download Presentation

William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COMPARING THE ACCURACY OF PERFORMING DIGITAL AND PAPER CHECKLISTS USING A FEEDBACK INTERVENTION PACKAGE DURING NORMAL WORKLOAD CONDITIONS IN SIMULATED FLIGHT William Rantz Western Michigan University Department of Aviation Sciences March 16, 2009

  2. Overview • Rationale & Purpose • Location & Duration • Frasca Flight Training Device • Participants • Paper-Digital Checklists & Flight Pattern • Dependent Variables • Independent Variables & Integrity of IVs • Experimental Design • Results • Inferential Results • Discussion & Future Research

  3. Rationale • Most common error cited in LOSA data • Observational data • 54% of errors(Helmreich et al., 2001) • Contributing factor to numerous accidents • Improper configuration of aircraft(NTSB, 1969, 1975, 1982, 1988a, 1988b, 1989, 1990, 1997, 1998, 2001, 2002, 2003a, 2003b, 2004a, 2004b, 2006, 2007a, 2007b, 2008a, 2008b, 2008c, 2008d) • Improper use of checklist (Adamski & Stahl, 1997 Degani, 1992, 2002; Diez, Boehm-Davis, & Holt, 2003; Federal Aviation Administration [FAA], 1995, 2000; Lautmann and Gallimore, 1987; Turner, 2001)

  4. Rationale • Digital Checklist Importance • Assertion: Errors prone to paper are reduced by digital format • Items skipped • Losing place when distracted • Incorrect switch selected • Item incorrectly confirmed complete • Excessive psychomotor workload fumbling with paper • Unreadable text due to low illumination • Subsequent checklist accomplished before critical flight phase • All checklists omitted • Enhanced flow of branched sequences-abnormal checklists • These errors not experimentally confirmed(Boorman, 2001) • Improved checklist performance using graphic feedback • Normal work load in simulated environment • Paper checklists(Rantz, Dickinson, Sinclair, Van Houten, 2009)

  5. Purpose • To examine whether pilots would complete airplane digital checklists more accurately when they received post-flight graphic and verbal feedback • No study compared the accuracy of traditional paper against emergent digital checklists • Only second study in aviation to attempt to increase checklist accuracy using experimental manipulation of IVs

  6. Location & Duration • The simulation laboratory is located in a hanger at WMU’s Aviation Education Center in Battle Creek, MI • Data collection took approximately 64 sessions • July 29, 2008 through March 11, 2009 • 256 flight trials

  7. Frasca 241 Flight Training Device –Cirrus SR20

  8. Frasca 241 Instructor Station Cirrus SR20

  9. Cirrus SR20 Primary Flight Display (PFD)

  10. Cirrus SR20 Multi Function Display (MFD)

  11. Cirrus SR20 Multi Function Display (MFD) Checklist

  12. Observation Area

  13. Participants • 6 WMU flight students • Males • Private Pilot Certificate • 125 minimum flight hours • Instrument rated • Average 186 total time flight hours • Average 80 hours total time in Cirrus Aircraft or FTD

  14. Digital Flight Checklist

  15. Paper Flight Checklist

  16. Null Hypotheses • 1) There is no intervention effect with either paper or digital checklists. • Tested at both the individual and group level • 2) There is no difference between paper or digital checklist accuracy during all phases.

  17. Main Dependent Variable • The number of paper checklist items completed correctly per flight • The number of digital checklist items completed correctly per flight

  18. Secondary Dependent Variable 1 • The percentage of total errors for each of the eight flight segments using the paper checklist during each experimental phase (baseline, feedback, and reversal) per participant • The percentage of total errors for each of the eight flight segments using the digital checklist during each experimental phase (baseline, feedback, and reversal) per participant

  19. Secondary Dependent Variable 2 • The percentage of baseline trials participants performed each of the paper checklist items incorrectly • The percentage of baseline trials participants performed each of the digital checklist items incorrectly

  20. Secondary Dependent Variable 3 • The percentage of baseline trials participants omitted paper checklist items • The percentage of baseline trials participants omitted digital checklist items

  21. Secondary Dependent Variable 4 • Percentage of timing errors participants performed paper checklist segments too early or too late • Percentage of timing errors participants performed digital checklist segments too early or too late

  22. Experimental Phases • Baseline • Only technical feedback of flight performance was given • Checklist Graphic Feedback & Vocal Praise • Technical feedback of flight performance • Graphic feedback on the total number of checklist items completed correctly per flight • Graphic feedback on the number of items completed correctly, completed incorrectly, and omitted for each of the eight flight segments per flight • Vocal praise for any improvement • Reversal • Only technical feedback of flight performance was given • 60-90 Day Data Probe • No Feedback was given

  23. Technical Feedback 23

  24. Graphic Feedback Received by Participant:Total Checklist Items Correct

  25. Graphic Feedback Received by Participant:Flight Segments (Total-Correct-Incorrect-Omitted)

  26. Inter Observer Agreement • Inter-observer agreement for correct and incorrect items was an average of 95% with a range of 79% to 100%. • Inter-observer agreement for omitted items was an average of 97% with a range of 63% to 100%.

  27. Integrity of the IV • Technical flight and checklist feedback were read from prepared scripts • Participants were asked to initial the technical flight diagrams and the checklist feedback graphs and returned to the experimenter. • Integrity of IV = 100%

  28. Experimental Design • A multiple baseline with reversal design across paired individuals • Initial phase change occurred when performance was judged as stable upon visual inspection • Reversal phase change occurred within 3 consecutive (paper or digital) trials exceeded 95% items correct • Probe phase occurred between 60-90 days past last reversal trial

  29. Results • Individually and grouped-all participants using both paper and digital checklists increased performance accuracy over baseline when post-flight checklist feedback and praise was added • Improvements in performance remained near perfect during intervention withdrawal • Improvements declined slightly within the 60-90 day probe period • Results were statistically significant

  30. Results-Figure 5continued • Average percentage of paper checklist items completed correctly increased from 38% during the baseline phase to 90% during the intervention phase • Average percentage of digital checklist items completed correctly increased from 39% during the baseline phase to 89% during the intervention phase

  31. Results-Figure 5continued • The average percentage of paper checklist items completed correctly was nearly 100% during the return to baseline condition • The average percentage of digital checklist items completed correctly was 99% during the return to baseline condition

  32. Results-Figure 5continued • The average percentage of paper checklist items completed correctly was 97% during the probe condition • The average percentage of digital checklist items completed correctly was 96% during the probe condition • 3% decrement in paper performance after 2-3 months • 4% decrement in digital performance after 2-3 months

  33. Results 33

  34. Results 34

  35. Results 35

  36. Results 36

  37. Results 37

  38. Results 38

  39. Results-Paper Checklist Segments-Figure 6 • During all flights using paper checklists, 2,451 total errors were observed • The average percentage of paper checklist segment errors was highest for the normal take-off segment (75%, range = 56.67% - 84.44%) • The average percentage of paper checklist segment errors was lowest for the after landing segment (50%, range = 0% -100%) • Please see Figure 6 hand out

  40. Results-Digital Checklist Segments-Figure 7 • During all flights using digital checklists, 2,562 total errors were observed • The average percentage of digital checklist segment errors was highest for the normal take-off segment (71%, range = 56.67% - 88.89%) • The average percentage of digital checklist segment errors was lowest for the after landing segment (50%, range = 1.85% -100%) • Please see Figure 7 hand out

  41. Results-Figure 6 & 7 Summation • Generally, the percentage of errors by flight segment for both paper and digital checklists varied across participants and flight segments • Errors decreased considerably for all participants during intervention. • Errors were very low during reversal • Errors increased slightly after 60 days

  42. Results-Paper Checklist Total Incorrect Items-Table 1 • Percentages of incorrect items that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded if the percentage of error was 50% or greater for four or more participants • The highest frequency of errors, across nearly all participants occurred for nine items in the before take-off segment: ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PIOTOT HEAT, NAV LIGHT, AND LANDING LIGHT. (AVG 86% n=9) • Four items in the normal takeoff segment were problematic for nearly all participants: POWER LEVER, ENGINE PARAMETERS, BRAKES, ELEVATOR CONTROL. (AVG 92% n=4) • Please see Table 1 hand out

  43. Results-Digital Checklist Total Incorrect Items-Table 2 • Percentages that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded if the percentage of error was 50% or greater for four or more participants • The highest frequency of errors, across nearly all participants occurred for the same nine items in the before take-off segment: ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PITOT HEAT, NAV LIGHT, AND LANDING LIGHT. (AVG 91% n=9) • Four items in the normal takeoff segment were problematic for nearly all participants: POWER LEVER, ENGINE PARAMETERS, BRAKES, ELEVATOR CONTROL. (AVG 84% n=4) • Please see Table 2 hand out

  44. Results-Paper Checklist Total Omitted Items-Table 3 • Percentages of omitted items that are 50% or greater are shaded for each participant. Also, the checklist item name is shaded if the percentage of omission was 50% or greater for four or more participants • There were no shaded omission errors for any checklist item either paper or digital • There were no shaded omission errors for any participant either paper or digital • No omitted item resulted in a crash or incident • There were only a random selection of omission items across many segments that P1 (AVG 19%), P2 (AVG 13%), and P4 (AVG 18%) did not perform. • Please see Table 3 hand out

  45. Results-Digital Checklist Total Omitted Items-Table 4 • Percentages of omitted digital checklist items that are 50% or greater are shaded for each participant. Also, the digital checklist item name is shaded if the percentage of omission was 50% or greater for four or more participants • There were no shaded omission errors for any checklist item either paper or digital • There were no shaded omission errors for any participant either paper or digital • No omitted item resulted in a crash or incident • P1 omitted a high percentage (AVG 24%) of items during the CRUISE, DESCENT, and AFTER LANDING segments • P4 omitted a high percentage (24%) randomly across several segments • P2 omitted a high percentage (15%) of the same items cited as done incorrectly in both the paper and digital during the before takeoff segment (ALTERNATOR, PITOT HEAT, NAV LIGHT, LANDING LIGHT, ANNUNCIATOR LIGHT, VOLTAGE, PITOT HEAT, NAV LIGHT, AND LANDING LIGHT) • Please see Table 4 hand out

  46. Results-Paper Checklist Segment Timing Errors in Baseline-Figure 8 • Descent (43%) • Before Landing (39%) • Climb (38%) • Please see Figure 8 hand out

  47. Results-Digital Checklist Segment Timing Errors in Baseline-Figure 9 • Before Landing (42%) • Descent (39%) • Climb (22%) • Please see Figure 9 hand out

  48. Individual Inferential Statistical Analysis • General time-series intervention regression modelingHuitema and McKean (1998, 2000a,b) and McKnight, McKean, and Huitema (2000) • Bootstrap based time-series regression method estimate parameters of individual’s behavior • AVG Baseline, Beta2, Beta3, AVG remaining Intervention, AVG Reversal, AVG Probe

  49. Beta Source 49

  50. Group Inferential Statistical Analysis • The purpose of this analysis was to provide an overall evaluation of the effects of the interventions for the group of six pilots over time • Previous time-series regression estimates parameters of individual’s behavior are used as DVs • One sample T-test used to evaluate hypothesis (no difference in intervention or phase changes) • Ho: AVG mean of Beta1 = 0

More Related