320 likes | 327 Views
This workshop focuses on training raters to ensure safety in aviation language proficiency evaluations. Follow ICAO recommendations for personnel selection, certification, reliability, training, and evaluation. May 24-26, 2011 in St-Petersburg, Russia.
E N D
Associate Member Raters Training as a Safety Concern ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011 Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters Training as a Safety Concern - RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OFRATER’S ADEQUACY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Aviation English test results are felt around the globe Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 3 instead of Level 4: Loss of job and/or occupation Financial losses for AO or ANSP Market losses Redistribution of air transportation services Lack of confidence for test-takers Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 4 instead of Level 3: Aviation safety is compromised Degradation to Level 2 Overconfidence for test-takers Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Annex 1; Doc 9835; ICAO Rated Speech Samples CD; ICAO Circ 318 Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Operational and/or language specialist expertise Level 5 or Level 6 proficiency Familiarity with Aviation English RATERS CODE OF ETHICS Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Theory of language testing • ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835,Circ 318 • The ICAO Proficiency Rating Scale and holistic descriptors • Correlating speech samples with the ICAO LPR Scale Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Certification by national Civil Aviation Authority or by international aviation organization is mandatory Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Is geographical principle necessary? Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • Are quotas necessary? Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different raters but same results Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different times but same results Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Administrative measures to marginal raters Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Practical training Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 • RATER AS A DECISION MAKER • SELECTION OF PERSONNEL • AB INITIO TRAINING • RATERS CERTIFICATION • INTER-RATER RELIABILITY • INTRA-RATER RELIABILITY • RASCH ANALYSIS • RECURRENT TRAINING • EVALUATION OFRATER’S ADEQUACY Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Thank you Raters Training as a Safety Concern