1 / 22

Standardisation in Language Rating

Standardisation in Language Rating. Henry Emery Vice President, ICAEA. Overview:. The need for regulation A mechanism for international rater standardisation On line administration Costs Challenges. Training. Training. Test. Training. Test. Training. Test. Training.

nika
Download Presentation

Standardisation in Language Rating

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Standardisation in Language Rating Henry Emery Vice President, ICAEA

  2. Overview: • The need for regulation • A mechanism for international rater standardisation • On line administration • Costs • Challenges

  3. Training Training Test Training Test Training Test Training Ultimate responsibility Test Results

  4. English language proficiency language and linguistics First language exposure to international accents aviation operations exposure to a particular accent language assessment rater training

  5. Intra-rater reliability

  6. Inter-rater reliability

  7. Inter-rater reliability

  8. ICAO Document 9835, Section 6.7.4 Raters must • understand the criteria and the context in which the criteria occur • agree upon the standardised procedures for the implementation of the criteria • inspire trust and gain the confidence of the community and the candidates

  9. ICAO Document 9835, Section 6.7.4 “A body of well-informed and experienced raters must be formed”

  10. AALR An Association of Aviation Language Raters

  11. Logo, Website and Letterhead Support from ICAO Support from Civil Aviation Authorities Sponsorship Phase 1 – Pre - Operational December 2008

  12. Gather speech samples Develop On Line System Rate speech samples Standardise ratings (Many Facet Rasch Analysis) Produce rationales Phase 2 : Pre - Operational December 2009

  13. AALR Phase 3 - Operational 1. Rated samples 2. Ratings 3. Certification January 2010 – May 2011

  14. Executive Officers System Administrators Senior Raters Certification Officers Prospective Raters Certified Raters Stakeholder Organisations Users of the AALR website Users of the On Line System • The general public • Participant testing organisations • Association Sponsors

  15. AALR Website Executive Officer(s) AALR Online System (OLS) On line system (OLS) Participant Testing Organisations The general public Association Sponsors Webmaster, System Administrator Senior Language Rater Certification Officer(s) Stakeholder Organisations Prospective Raters Certificated Raters

  16. Start up and operational costs to May 2011 £103,260 Estimated Costs Fees generated to May 2011 £22,500 Sponsorship required to May 2011 £80,760 Average* operational cost per year £30,040 Fees generated per year £32,500 Balance of account per certification period +£4,920

  17. Questions Name Relationship with ICAEA Disclaimers Funding Delegation

  18. On-going maintenance Test specification Design and construction Scoring and reporting Trialing Test administration Personnel training Analysis and evaluation

  19. On-going maintenance Test specification Design and construction Scoring and reporting Trialing Test administration Personnel training Analysis and evaluation International Rater Standardisation

  20. Questions Name Relationship with ICAEA Disclaimers Funding Delegation

  21. Questions Testing organisations Senior Raters Support / Endorsement Academic support / oversight End user charges

  22. AALR Association of Aviation Language Raters Comments : henry@emery-roberts.co.uk

More Related