"A system that trains the evaluator"

1. From NOTECHS to a New System ofObserving Behavioural Markers as aBasis for Assessing Pilots' Performance "A system that trains the evaluator" Cpt. Harry Neb, Lufthansa German Airlines FRA PC/L Human Factors Operational Input

2. LH Safety Study (1997) LH Interpersonal Competence NOTECHS JAA LH CRM-Markers (1992) External Information (scientific, other airlines, RAF, etc.) Influences and Sources FRA PC/L Human Factors

3. Influences and Sources JAA FRA PC/L Human Factors

4. Frequency Distributionby Event-Configurations 9.1% 46.2% 30.8% 14.5% FRA PC/L Human Factors

5. Aggravating Social Factors contributed to 71% of events. Deficiencies inInvolvement • Information management 67.2% • Quality of communication 52.3% • Social climate 27.2% FRA PC/L Human Factors

6. FRA PC/L Human Factors

7. Elements Communication Generally, communication includes information transfer and social aspects. Crew members share information, and assure reception and understanding. Suggestions of other crew members are considered, even if one does not agree. Ambiguities and uncertainties are announced. Workload Management Leadership and Teamwork Crew members clearly prioritise operational tasks and distribute them appropriately. Available external and internal resources are used to accom-plish task completion in good time. Stress and error are inherent factors of flight, and crew members aim to minimise their negative effects. Led by the commander, the crew achieves a safe and efficient performance in a climate that is rational and free of intimidation. Social interaction conflicts have to be addressed and managed. Every crew member takes initiative to be an active and constructive part of the team. InterpersonalCompetence Situation Awareness and Decision Making Crew members recognise and anticipate factors affecting the flight. After these factors are evaluated, they choose the appropriate course of action. To achieve a favourable outcome, crew members actively monitor execution and development of the situation. FRA PC/L Human Factors

8. Communication Atmosphere encourage open and honest communication achieve a positive first impression listen actively consider suggestions Information Transfer share information assure reception assure understanding Information Management clearly state plans and intentions announce ambiguities announce uncertainties speak frankly about problems within the crew Descriptors FRA PC/L Human Factors

9. Descriptors Leadership & Teamwork Command- ability take the lead of the crew as commander establish goals, control outcome, and correct consider condition of others Team Ability act as a constructive member of the team take initiative encourage others to co-operate support others seek ideas and views from others present own point of view provide appropriate feedback propose alternative ideas if appropriate Conflict Management address and manage conflicts achieve rational climate avoid intimidation adopt assertive behaviour if appropriate and persist until attention of others is gained or corrective action taken accept appropriate criticism avoid competition between crew members FRA PC/L Human Factors

10. Workload Management Task prioritise operational tasks distribute tasks appropriately complete tasks in good time use external and internal resources Time plan ahead allocate time to tasks appropriately Stress and Error aim to minimise negative effects of stress aim to minimise effects of error Descriptors FRA PC/L Human Factors

11. Descriptors Situation Awareness and Decision Making Preparation act with respect to time available avoid distractions anticipate factors affecting the flight recognise factors affecting the flight Processing evaluate factors affecting the flight choose appropriate course of action monitor execution monitor development of the situation apply FOR-DEC for complex decisions F-Facts, O-Options,R-Risks & Benefits, D-Decision,E-Execution, C-Check Interaction involve others in the process discuss discrepancies FRA PC/L Human Factors

12. Evaluation A New Concept. Why?

13. Evaluation Prerequisites • Appropriateness and Relevance • Standardization of contents and execution • Validity: Accuracy of correspondence with the relevant learning goals • Reliability: Constant accuracy and correspondence with comparable results • Objectivity: Same results with application by different evaluators • Transparency and Reasonableness: No traps and tricks • Fairness: Equal treatment of candidates • from: K. Steininger, Handbuch der Flugpädagogik FRA PC/L Human Factors

14. Present Status • Description of performance in the words of the individual evaluator • Subjective view due to different priorities • Interpretation of assessment by the training management • CRM criteria (behavioural markers) not integrated FRA PC/L Human Factors

15. Present Status contd. • Quality description in two steps: • proficient • repeat • Trend not recognizable • Flaws in the system not identifiable FRA PC/L Human Factors

16. Goal (Highlights) For all assessments Uniformity Objectivity Take calculated measuresat the earliest possible stagefor any performance below standard FRA PC/L Human Factors

17. Uniformity • Demand the same performance under the same circumstances • Ask the same questions • Observe the same defined behaviour • Treat each trainee equally FRA PC/L Human Factors

18. Goal (Highlights) For all assessments Uniformity Objectivity Take calculated measuresat the earliest possible stagefor any performance below standard FRA PC/L Human Factors

19. Objectivity • Application by different evaluators leads to the same result FRA PC/L Human Factors

20. Goal (Highlights) For all assessments Uniformity Objectivity Take calculated measuresat the earliest possible stagefor any performance below standard FRA PC/L Human Factors

21. Expected Outcome • Assess quality to manage quality • Identify trends to assure (maintain or improve)quality with • Trainee • Instructor • System FRA PC/L Human Factors

22. Identified Necessary Actions • Define detailed observable rating criteria=> Description of company standard • By observing frequencies of behaviourthe progress and the absolute performance to be described • Lead instructors to observelargely the same • Make transparent for the trainee FRA PC/L Human Factors

23. Identified Necessary Actions • Differentiate quality of performance • Define behavioural markers to describe theinterpersonal competence FRA PC/L Human Factors

24. Quality Assurance for: TRAINEE by providingwell aimed support for cases of performance below standard{below satisfactory}(because identifiable) FRA PC/L Human Factors

25. Quality Assurance for: TRAINER(Evaluator) by enabling comparable assessments(for objectivity) FRA PC/L Human Factors

26. Quality Assurance for: SYSTEM to improve the training process(by discovering weak spots) FRA PC/L Human Factors

27. ! Goal (Trainer): • No substantial change ofmethodical and didactic means for training(The system does not intervenein the personal style of training) FRA PC/L Human Factors

28. Definition Trainer: "A trainer is an employee trained in training who is instructed by the traineeship provider to train trainees." Hamburg Senate "Definitions" FRA PC/L Human Factors

29. Goal (Trainer): Keep the sequence: • Observe • Note • Evaluate FRA PC/L Human Factors

30. Observation • Prerequisite for evaluation isobservation ofQuantity and Qualityof behaviour how often what FRA PC/L Human Factors

31. What Defines Quality Specified OBSERVATION CRITERIA (i.e. behavioural markersresp. observable behaviour) QUALITY FRA PC/L Human Factors

32. Criteria Thereforeobservation criteria have to be: • relevant • extensive • observable FRA PC/L Human Factors

33. Structure 9 defined phases of flight(from "preparation" to "leaving aeroplane") + general descriptors of the areas technical procedural interpersonal FRA PC/L Human Factors

34. Break FRA PC/L Human Factors

35. What Next? • Principles of Behaviour Assessment in NOTECHS/JARTEL • Development of Lufthansa‘s New Behaviour Marker System • Crosslinking NOTECHS to LH Interpersonal Competences • The Implementation Case: Lufthansa‘s Evaluation Method • Usability Study • Summary