1 / 32

Raters Training as a Safety Concern

Associate Member. Raters Training as a Safety Concern. ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011. Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director. Associate Member. ICAO LPRI Workshop, St-Petersburg, Russia, 2011.

kirby-robbins
Download Presentation

Raters Training as a Safety Concern

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Associate Member Raters Training as a Safety Concern ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011 Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director

  2. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern - RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OFRATER’S ADEQUACY

  3. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  4. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Aviation English test results are felt around the globe Raters Training as a Safety Concern

  5. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 3 instead of Level 4: Loss of job and/or occupation Financial losses for AO or ANSP Market losses Redistribution of air transportation services Lack of confidence for test-takers Raters Training as a Safety Concern

  6. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 4 instead of Level 3: Aviation safety is compromised Degradation to Level 2 Overconfidence for test-takers Raters Training as a Safety Concern

  7. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  8. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications Raters Training as a Safety Concern

  9. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Annex 1; Doc 9835; ICAO Rated Speech Samples CD; ICAO Circ 318 Raters Training as a Safety Concern

  10. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Operational and/or language specialist expertise Level 5 or Level 6 proficiency Familiarity with Aviation English RATERS CODE OF ETHICS Raters Training as a Safety Concern

  11. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  12. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Theory of language testing • ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835,Circ 318 • The ICAO Proficiency Rating Scale and holistic descriptors • Correlating speech samples with the ICAO LPR Scale Raters Training as a Safety Concern

  13. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale Raters Training as a Safety Concern

  14. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  15. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Certification by national Civil Aviation Authority or by international aviation organization is mandatory Raters Training as a Safety Concern

  16. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Is geographical principle necessary? Raters Training as a Safety Concern

  17. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Are quotas necessary? Raters Training as a Safety Concern

  18. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  19. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different raters but same results Raters Training as a Safety Concern

  20. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  21. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different times but same results Raters Training as a Safety Concern

  22. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  23. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Raters Training as a Safety Concern

  24. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Raters Training as a Safety Concern

  25. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Administrative measures to marginal raters Raters Training as a Safety Concern

  26. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  27. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters Raters Training as a Safety Concern

  28. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Raters Training as a Safety Concern

  29. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Practical training Raters Training as a Safety Concern

  30. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern

  31. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN Raters Training as a Safety Concern

  32. Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Thank you Raters Training as a Safety Concern

More Related