1 / 16

Walde k Martyniuk ECML, Graz, Austria

Walde k Martyniuk ECML, Graz, Austria. Relating examinations to the CEFR – the Council of Europe Manual and supplementary materials. The CEFR and assessment. Potential Challenges Solutions Recommendations. CEFR and assessment: the potential. „ … the Framework can be used:

billymartin
Download Presentation

Walde k Martyniuk ECML, Graz, Austria

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Waldek Martyniuk ECML, Graz, Austria Relating examinations to the CEFR – the Council of Europe Manual and supplementary materials

  2. The CEFR and assessment • Potential • Challenges • Solutions • Recommendations

  3. CEFR and assessment:the potential „ … the Framework can be used: • for the specification of the content of tests and examinations • for stating the criteria to determine the attainment of a learning objective (…) • for describing the levels of proficiency in existing tests and examinations thus enabling comparisons to be made across different systems of qualifications” (CEFR: 178)

  4. CEFR and assessment: the potential • 6 levels of proficiency defined largely in relation to empirically derived difficulty estimates based on stakeholder perceptions of what language functions expressed by Can-Do statements can be successfully performed at each level • „least arbitrary sequence of scaled proficiency descriptors available to us at the moment” (Weir 2004)

  5. CEFR and assessment: challenges • Test development and analysis • Relating examinations • Validating claims • QMS

  6. Using CEFR for test development and analysis • „the scales are premised on an incomplete and unevenly applied range of contextual variables/performance conditions (context validity); • little account is taken of the nature of cognitive processing at different levels of ability (theory based validity); • activities are seldom related to the quality of actual performance expected to complete them (scoring validity); • the wording for some of the descriptors is not consistent or transparent enough in places for the development of tests.” (Weir 2004)

  7. CEFR and assessment:the limitations • NOT a harmonisation tool: “We have NOT set out to tell practitioners what to do or how to do it. We are raising questions not answering them. It is not the function of the CEF to lay down the objectives that users should pursue or the methods they should employ” (CEFR: xi) • NOT a theory of language development: Scales describe learning outcomes, learner behaviours, not theinvisible processes involved. • NOT a test specification: Scales and lists can be consulted when drawing up a taskspecification (Ch. 4) or defining assessment criteria (Ch. 5) but needreference to detailed specifications for language & context. (North 2004)

  8. CEFR and assessment:solutions • Reference Level Descriptions • Grids for the analysis of test items and tasks • Manual for relating examinations to the CEFR + supplement • Benchmarking conferences • Benchmarked test items/tasks and performances • Guidelines for examiners and test developers (revised)

  9. CEFR and assessment:recommendations • Main types of assessment: • Placement • Formative • Summative • Main modes of assessment: • Self-assessment • Teacher assessment • Test (North 2005)

  10. Manual for relating language examinations to the CEFR • Drafted by an authoring group in 2003 • Piloting the Draft: 2004 - 2007 • Revising and publishing: 2009 • Case study publication: 2009 • Standard setting publication: 2009

  11. Suggested Procedures • Familiarisation with the CEFR • Linking on the basis of specification of examination content • Standardisation and Benchmarking • Standard setting • Empirical validation: checking that exam results relate to CEFR levels as intended

  12. Reference Materials • Reference Supplement • Videos and DVDs with calibrated sample spoken performances • CD-ROMs with calibrated sample items and tasks for reading and listening • Calibrated sample written tasks and performances (web page based) • Grids for the analysis of test items, tasks and performances

  13. The Manual 2009 Chapter 1: The CEFR and the manual Chapter 2: The Linking Process Chapter 3: Familiarisation Chapter 4: Specification Chapter 5: Standardisation Training and Benchmarking Chapter 6: Standard setting (panel based approach) Chapter 7: Validation Appendices: A: forms and scales for chapter 4 B: content analysis Grids (chap 4) C: forms and scales for chapter 5

  14. The Manual 2009 • Appendices: • A: forms and scales for chapter 4 • B: content analysis Grids (chapter 4) • C: forms and scales for chapter 5 • Further material: • Standard setting and standard maintaining (scalar perspectives)

  15. The Reference Supplement • Section A: Summary of linking process • Section B: Standard Setting • Section C: Classical Test Theory • Section D: Qualitative Analysis Methods • Section E: Generalisability Theory • Section F: Factor Analysis • Section G: Item Response Theory • Section H: Test Equating

  16. All materials available for downloading from:www.coe.int/langorwww.coe.int/portfolioThank you for your kind attention!

More Related