1 / 16

University of Michigan English Language Institute Testing and Certification Division

14 th International GALA conference, Thessaloniki, 14-16 December 2007. Behavioural scales of language proficiency: insights from the use of the Common European Framework of Reference Spiros Papageorgiou. University of Michigan English Language Institute

wayne
Download Presentation

University of Michigan English Language Institute Testing and Certification Division

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 14th International GALA conference, Thessaloniki, 14-16 December 2007 Behavioural scales of language proficiency: insights from the use of the Common European Framework of Reference Spiros Papageorgiou University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  2. Outline • Background • Aims • Data collection • Data analysis • Results • Implications University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  3. Background • Advent of the CEFR: increased interest in behavioural scales of language proficiency • Using the CEFR scales: Problems • Designing test specifications (Alderson et al., 2006) • Measuring progression in grammar (Keddle, 2004) • Describing the construct of vocabulary (Huhta & Figueras, 2004) • Designing proficiency scales (Generalitat de Catalunya, 2006) University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  4. Background(2) • Using the CEFR scales: Criticism • Equivalence of tests constructed for different purposes (Fulcher, 2004b;Weir, 2005) • Danger of viewing a test as non valid because of not claiming relevance to the CEFR (Fulcher, 2004a) • Progression in language proficiency not based on SLA research but on judgements by teachers (cf. North 2000; North & Schneider 1998) University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  5. Aims of the study • Investigation of three research questions: • Can users of the CEFR rank-order the scaled descriptors in the way the appear in the 2001 volume? • If differences in scaling exist between the users of the CEFR and the 2001 volume, why does this happen? • Can training contribute to more successful scaling? University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  6. Data collection • 12 users of the scales acting as judges in relating two language examinations to the CEFR • Data collected during Familiarisation sessions described in the Manual for relating examinations to the CEFR • Part of a doctoral thesis at Lancaster University (Papageorgiou, 2007) and a research project at Trinity College London • Task: sort descriptors into the six levels University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  7. Data collection (2) University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  8. Data analysis • Analysis: FACETS Rasch computer program • 3 facets: descriptors-raters-occasions • Rank-ordering of elements of facets on a common scale • Fit statistics (Bond and Fox, 2001; McNamara, 1996) • Overfit: too predictable pattern • Misfit: more than expected variance • Acceptable range of fit statistics • Descriptors: .4-1.2 (Linacre & Wright, 1994) • Raters: .5-1.5 (Weigle, 1998) University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  9. Results: Writing Levels A1-B1 University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  10. Results: Writing Levels B2-C2 University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  11. Results: Raters University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  12. Results: Occassions University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  13. Results: Correlations Correlations of scaling between the judges and the CEFR volume University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  14. Summary of results • Trained judges perceived language ability as intended in the CEFR • Almost identical scaling • Cut-offs between B2-C1 and C1-C2 unclear • Competences other than linguistic: misfitting descriptors • Unclear and inconsistent wording resulted in level misplacement by the judges • Mixed effect of training University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  15. Implications of findings • Common understanding of the construct in the CEFR scales can be achieved, but • How valid is it to claim that a test is linked to B2 instead of C1 and C1 instead of C2? • How can sociolinguistic and strategic competences be tested in relation to the CEFR? • Can SLA research help better understand these issues? University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

  16. Contact details Spiros Papageorgiou University of Michigan English Language Institute 500 East Washington Street Ann Arbor, MI 48104-2028 USA spapag@umich.edu University of Michigan English Language Institute Testing and Certification Division www.lsa.umich.edu/eli

More Related