1 / 18

Research & Technology Europe

Usability Study of Lufthansa’s Performance Evaluation System Hans-Juergen Hoermann, PhD Boeing Research & Technology Europe RAeS, April 30, 2004. Research & Technology Europe. Overview. Introduction Design of the usability study

finley
Download Presentation

Research & Technology Europe

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Usability Study ofLufthansa’s Performance Evaluation SystemHans-Juergen Hoermann, PhDBoeing Research & Technology EuropeRAeS, April 30, 2004 Research & Technology Europe

  2. Overview • Introduction • Design of the usability study • Quantitative results concerning the practicability and training benefits of the new Proficiency Reporting Form (PPR) • Summary of written user comments • Recommendations for a train-the-trainer course • Conclusions

  3. LH Flight safety survey Update of performance criteria for LH flight-crew members New concepts for proficiency reporting forms JAA requirements (NOTECHS/JARTEL) Coinciding Inputs From Different Initiatives

  4. Based on the LH Competence System with the three areas: technical, procedural, and interpersonal Frequency scale for quantitative observations Four-point grading scale for quality of performance Competence codes for feedback and further training recommendations Training goals explicitly included Special Features of the New PPR

  5. User Input & Feedback Form: 15 yes/no items with plenty of room for free comments, concerning ... Structure, layout and contents of the new Evaluation System Practicability and use of different scales Clearness and adequacy of results Value of Comp. Codes for debriefing and training Usability Study

  6. Design and procedure: N = 45 TCs from 3 LH fleets One-day introductory course Telephone hotline 4 months trial period with three forms N = 20 returned with full data Based on training events with 27 captain candidates Follow-up crosscheck of the revised form within another LH subsidiary (N = 13 TCs) Usability Study

  7. User feedback in form of: 76 (98) specific comments 13 (18) general comments Usability Study

  8. Structure, Layout, and Contents 85 - 90% Sufficiently clear? Criteria complete? 80% Comments: no gaps, suggestions for reducing items, clear structure significantly reduces efforts to fill out PPR

  9. Practicability and Use of Scales 95% Famil-course OK? 94% Freq count practicable? Req. time justifiable? 85%

  10. Practicability and Use of Scales Comments: The clear structure makes it less time-consuming than before, Difficult to handle the papers especially when also active as CM Quantity is not always equivalent with quality

  11. Clearness and Adequacy of Results 65% Unambiguous? 75% 90% Definite and adequate? Different to previous? 15% Comments: Scale anchor „Always“ is too strict. Minor errors would lead to a downgrading of otherwise good performance

  12. Use of Comp. Codes for Training & Debrief 90% Useful for training? 50-60% Facilitate debrief? Comments: Training goals more transparent for trainer and trainee. Weaknesses can be identified earlier. CCs helpful for finding adequate training recommendations.

  13. Distribution of Comp. Codes in the Study

  14. Observation is determined by more specific criteria instead of the overall impression Debriefing could become a little longer but also more objective and supportive All in all, a little more effort for the benefit of a more adequate evaluation and feedback of the candidate‘s performance More objective because of more observation items (CLH) Because of the versatility of the observation items the form is good for a fair and more standardised evaluation (CLH) General Comments

  15. User feedback has provided valuable insights for further revisions of the PPR and contents of the familiarization course Acceptance of deviation from absolute perfection (100%  99+%) High degree of acceptance for the integration of interpersonal skills, training should precede checking Full transparency of training goals offers better opportunity for self-evaluation and preparation of trainees A long-term evaluation study is scheduled to detect trends and tailor future training methods Conclusions

  16. In addition to the required training of instructors and examiners it is recommended to have at minimum A two-day training session with a trainee-trainer ratio not exceeding 12:1 Timed in vicinity to the first training mission of the new instructor/examiner A thorough briefing on the respective non-technical skills concept including categories, elements, and desired/undesired behaviour anchors A thorough briefing on the rating scale, the evaluation criteria (cut-offs), and the forms A practical training session with specific video examples for each of the evaluation categories. This training includes feedback and discussion. Guidance material and practical training sessions of how to conduct a constructive debriefing on non-technical skills. Both in examiner and examinee roles. Regular refresher for re-calibration Written material, including definitions and criteria should be available and agreed throughout the company Training Course for NOTECHS Examiners

  17. Finally, we arrived at the summary… Lufthansa‘s Basic Competence System

  18. What is left? BR&TE • Principles of Behaviour Assessment in NOTECHS/JARTEL • Development of Lufthansa‘s New Behaviour Marker System • Crosslinking NOTECHS to LH Interpersonal Competences • The Implementation Case: Lufthansa‘s Evaluation Method • Usability Study • Summary

More Related