Dr Jeremy Mell Head of Language Studies, ENAC, Toulouse, France Member PRICE SG, ICAO

1. Language proficiency evaluation(testing)Selection and development of language tests in the aviation context Dr Jeremy Mell Head of Language Studies, ENAC, Toulouse, France Member PRICE SG, ICAO jeremy.mell@enac.fr ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

2. Options for assessing language proficiency for aviation • informal observations • « line checks », « inspections » • classroom assessments • formal language tests • direct/semi-direct/indirect • simulated language use • most direct • integrative: performance samples are matched to rating scales • paper and pencil OR screen and mouse tasks • most indirect • discrete items/skills: favour numerical scores • but « juggling with only one ball »…. ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

3. Language tests • Definition • structured events or procedures • to elicit performances as samples of test-taker ’s language skills • in a standardised way • enabling reliable inferences to be made concerning his/her level of competence • and possibility of reproducing those skills at that level of competence consistently over time • adapted from Carrol (1968) and Douglas (2000) ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

4. Language testing • A well-developed domain of intellectual activity: • academic • body of research, cross-disciplinary links, international associations,... • codes of ethics, codes of practice • institutions of excellence, « chapels », « gurus », feuds • commercial • international testing service providers • general, academic, business,… • Attempts to link different rating scales/test results ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

5. Types of language tests • entry • placement • diagnostic • progress/achievement • proficiency ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

6. Purposes ab initio training: recruitment form level groups recurrent training: benchmarking a population form level groups Characteristics items chosen to cover a broad range of levels of general language formats reflect previous education scores tend to form a « bell curve » stakes high (recruitment) mid (benchmarking) Entry/placement tests ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

7. Purposes identify specific areas of skill/knowledge for improvement in subsequent training individualisation of training programmes Characteristics each item chosen to represent a single significant area of knowledge/skill overall score less important than analysis of right/wrong responses - focus on errors stakes: low Diagnostic tests ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

8. Purposes to measure effectiveness of a phase of learning to allow access by learner to next phase of learning Characteristics items chosen to closely reflect content and methodology of preceding training phase scores will often be interpreted with regard to average or pre-established norm may refer to intermediate levels (3,5; 4,5;…) stakes: low to medium Progress/achievement tests ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

9. Purposes to establish the competence of candidate to exercise language skills in operational conditions Characteristics items chosen to resemble real-world tasks overall scores are holistic: YES/NO stakes: very high Proficiency tests ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

10. Ab initio populations entry selection local education vs aviation training policies progress/achievement intermediate training objectives continuation/curtailment initial proficiency licensing Qualified populations placement/diagnostic benchmarking populations individual training needs acceptance by takers (progress/achievement) degree and speed proficiency renewal recurrent Testing needs of aviation ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

11. Language test quality • qualities of tests in general • validity • face validity • reliability • washback effect • practicality • conflicting qualities ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

12. Test design specifications (ICAO) • “As of 5 March 2008, aeroplane and helicopter pilots, air traffic controllers and aeronautical station operators shall demonstrate the ability to speak and understand the language used for radiotelephony communications to the level specified in the language proficiency requirements in the Appendix.” (Annex 1, 1.2.9.4) • formal evaluation • “An individual must demonstrate proficiency at level 4 in all categories in order to receive a level 4 score.” (Doc 9835, Manual on the Implementation of ICAO LPRs, 2.8.4) • profile evaluation ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

13. Tasks develop test specifications construct test evaluate test ensure ongoing test maintenance Qualifications operational expertise language test development expertise linguistic expertise item writers Test design teams ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

14. Tasks ensure advance access to test samples schedule test manage pre-test preparations manage test Qualifications knowledge of administration guidelines interlocutor expertise operational linguistic rater expertise (min 2 raters) Test administration teams ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

15. Rater/interlocutor qualifications • 3 basic qualifications • language level • aviation background + familiarity with ICAO LPRs • principles of language proficiency and language testing • suitable candidates • aviation personnel • language trainers • L1 background • native speakers(NS)/non-native speakers (NNS) • risk of familiarity with a given form of NNS spoken English ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

16. Institutional context • need for independent international oversight to ensure: • validity of testing procedures used • comparability of testing procedures and outcomes • testing outcomes are linked with personnel licensing • testing may not be in the sole hands of the ELT community • partnerships with CAA, service provider, airline,... • system cannot tolerate a high failure rate • waste of expensive training investment • staffing levels must ensure continuity of service • prior knowledge of test procedures for positive “washback” on training • trainee motivation and attitude, reassurance of candidates • development of appropriate training systems • BUT unsuitability of “cramming”, test item banks ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

17. A common test for ATC and pilots? • pilots and controllers are partners in R/T communication • but they are set apart by: • different interactive roles • complementary passive/active competencies • opportunity to use other job-related language uses to extend speech sample: • controller: telephone co-ordinations, report to supervisor, ... • pilot: pre-flight, intra-cockpit, cabin announcements, ground staff, …. • a possible solution? • common core test • job-specific components ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

18. time/space constraints dispersed locations (recurrent) irregular schedules and limited availability of test-takers (recurrent) availability of technologies test/re-test test components single event, integrative? different item designs to test separate skills? standardisation of results extensive trialling comparable conditions of test administration examiner training/auditing interlocutors raters test security secure storage and transfer of test materials multiple parallel versions Need for testing services ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

19. Appropriate technologies • voice-only • telephone, 2-way radio, language laboratories, training simulators • input delivery • analogue players, computer screen (multimedia) • performance storage and access • analogue recorders, computer sound files (rapid access) • speech recognition • standardise perception of intelligibility, save rating costs • BUT needs to filter all possible speech variations • no real interaction • « interactions » need to be human-human • initiate and maintain exchanges • deal with misunderstandings ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

20. Standardisation • operationalisation of the rating scale • speech samples • « can do » statements • rater training • initial • refresher • rating protocols • paired rating • multiple rating ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

21. NOT a safety issue, but raises concerns about: resources ( no re-test) NS/NNS distinctions Speakers may be: mother-tongue non-mother tongue intranational use non-intranational use Candidacy on basis of biographical criteria citizenship, educational background, residency, ... Protocols for assessment admissible evidence assessor qualification appropriate documentation Subsequent checking and oversight must address: adherence to standard ICAO phraseologies ICAO LPRs for intelligibility appropriateness Assessment of level 6 ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

22. Informal evaluations • Within the initial or continuing training process, language trainers can identify 3 categories of trainee: • well-below level 4 • approximately at level 4 (confirm by formal evaluation) • comfortably above above level 4 • …thus enabling informed decisions to be made on • possible access to professional functions • further training required • benchmarking of populations ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

23. Placement (entry) must relate to ALL 6 skills in ICAO scale must measure distances above and below level 4 may have diagnostic capability tasks/topics/input materials focus on general or job-related skills development requires some statistical validation Proficiency (exit) must relate to ALL 6 skills in ICAO scale must characterise test-takers as BELOW or AT levels 4, 5 or 6 tasks/topics/input material must reflect radiotelephony language competencies (including standardized phraseology) development requires strong statistical validation (high stakes) Formal tests for ICAO level 4 ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

24. Detailed specifications (proficiency) • provide a representative range of intelligible international accents as input for comprehension; • provide a professionally relevant format for candidates to display comprehension; • elicit an adequate continuous speech sample to test fluency/pronunciation; • provide a voice-only setting for “diadic” (2-person) interactions; • provide examples of routine and unexpected events in a work-related context; • allow the candidate to use basic grammatical structures creatively; • allow the candidate to demonstrate ability to paraphrase; • allow the candidate to change between rehearsed/formulaic speech and spontaneous interaction; • simulate unexpected events to create opportunities for misunderstanding. • EUROCONTROL/ENAC preliminary feasibility study 2004 ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

25. Controllers ab initio entry: EPT/FEAST(Eurocontrol) progress: APRO (ENAC, F) proficiency: PELA (Eurocontrol) recurrent placement: TNP (DGAC, F) proficiency: ELPAC (Eurocontrol) under development for 2007 Pilots ab initio proficiency: FCL 1.200 (DGAC, F) Other tests TEA (Mayflower, UK) TELLCAP (Russia) TOEFA (Peru), TELPA (IAES, Korea) TELAP (CA Flight University,China) RELTA (RMIT, Australia) …. Some existing specific tests ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006

26. TNP (DGAC, France) • developed by national language and subject-matter experts (controllers and ATC management) as a placement/diagnostic tool • 3 papers cover professional language content: • a written multiple choice test of language knowledge • a tape-mediated written test of listening comprehension (RT messages) • an individual oral interview using paper-mediated graphic and text input • expresses results in terms of the ICAO level chart • used since 1999 with qualified air traffic controllers: • benchmarking to establish national training needs • formulating periodic individualised language development and maintenance programmes in order to meet national qualification renewal requirements • NOT a proficiency test • lacks robust validation ICAO Regional Workshop on Language Proficiency Requirements Implementation; Paris, France, 6-7 September 2006