From Norm- to Standards-based assessment What role can the CEFR play?

1. CBSE-CAER International Conference 2014 Delhi-NCR, 28th and 29th April 2014 From Norm- to Standards-based assessmentWhat role can the CEFR play? Dr Claudia Harsch Centre for Applied Linguistics University of Warwick

2. Overview • Norm- and Criterion-Oriented Assessment • The Common European Framework of Reference (CEFR) and its use in assessment • Classroom-based assessment • Educational Monitoring • International Certificates • Transition from norm- to criterion-orientation: Chances and Challenges • Preparing students and teachers • Aligning exams to the criterion (CEFR) • The multi-lingual context

3. Norm-Orientation Criterion-Orientation

4. Comparison (e.g. Bachman & Palmer 2010, Douglas 2010) Norm-Orientation Criterion-Orientation Qualitative description of ... - what skills / competencies learners have acquired - how well learners perform in relation to learning outcomes Transparent expectations, goals, outcomes expressed in criteria Comparability via common goal posts • Quantitative score • Ranking learners • Comparing learners • Placement decisions • Grading decisions

5. The Common European Framework of Reference (CEFR)in criterion-oriented assessment Classroom-based assessment Educational Monitoring International certification

6. The CEFR • Instrument to enhance communication and mobility (language policy of the Council of Europe) • At core: Proficiency model with levels of proficiency • Description of learners’ abilities as positive ‘Can do’ statements • Framework allows qualitative description of learning outcomes and assessment results (rather than only numerical scores) –> reference criteria

7. A1 A2 B1B2 C1 C2 The CEFR levels

8. The CEFR scale system Communicative Competence Communicative Language Activities Communicative Language Competences Communicative Strategies Reception Interaction Grammar Orthography Production Vocabulary Pronunciation

9. Student with high proficiency CEF Proficiency Levels C2: Can understand a wide range of long and complex texts, appreciating subtle distinctions of style and implicit as well as explicit meaning. A1: Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required. Student with low proficiency

10. CEFR in Classroom-based assessment • Defining learning outcomes, develop curriculum using CEFR descriptors • Self-assessment: via the self-assessment grid or the European Language Portfolio • Peer-assessment via common criteria, to be derived and specified on basis of e.g. grid or CEFR descriptors • Teacher assessment via the CEFR scales, need to specify for particular performances • Advantage of common and transparent goals, criteria and levels, shared expectations • Possibility of comparing different views on learners

11. CEFR in Educational Monitoring • CEFR as external standard / criterion, e.g. National Educational Standards (NES) Germany: • Defining learning outcomes /competencies expected at the end of secondary schooling in relation to CEFR levels and descriptors • Learning outcomes operationalised in standards-based tests • Tests formally aligned to the CEFR • Test results reported on competency scale (like PISA), accompanied by qualitative descriptions (CEFR, NES) • Advantage of linking external monitoring to internal teacher-/student-assessment

12. CEFR in International Certification • CEFR as common proficiency framework • International tests aligned to CEFR levels • Test results/reports: scores and CEFR levels • Comparability: CEFR as common reference point • Challenge: “Is my B1 your B1?” • High-stakes decisions, e.g. university admission • Political agenda, e.g. visa and immigration decisions (ethical?)

13. Transition from norm- tocriterion-based assessment Preparing teachers and students Aligning exams to the CEFR The Multi-lingual Context

14. Preparing teachers and students • Traditional focus on ranking, competition, errors, correct answers • Now: shift towards criteria, outcomes, skills • Focus on what learners can do with their language, whether they can perform certain communicative tasks • Requires willingness and readiness for change • Requires training resources

15. Preparing teachers and students • Teacher Training: Familiarity with CEFR • Document available online: www.coe.int/lang-CEFR • Online training: www.helsinki.fi/project/ceftrain • Teacher Training: Familiarity with rating approaches, self- and peer-assessment, Portfolio http://www.coe.int/t/dg4/education/elp/Default_EN.asp • Students: Introduction to self-assessment and Portfolio by their teachers, familiarity over time • Classroom: Developing a culture of peer-assessment, supportive and honest but respectful feedback

16. Aligning Exams to the CEFR • Manual to support such endeavourshttp://ww.coe.int/t/dg4/linguistic/manuel1_en.asp • Several steps to follow – resources needed • Range of standard setting methods available – expertise needed (e.g. Cizek et al 2007) • Human judgement – uncertainties and biases • Research projects ongoing to • report exam alignments in a transparent way • examine most feasible ways of alignment (e.g. Eckes 2012, Figueras et al. 2005, Harsch et al. 2010)

17. The Multi-Lingual Context • Specific multi-lingual skills, learning outcomes, expectations • Not covered in the CEFR • Possibility to define such a set of multi-lingual skills for the Indian context • Relevance • Transparency • Added value: setting an example for other multi-lingual contexts

18. Thank You for Your Attention Claudia Harsch C.Harsch@warwick.ac.uk

19. Selected References Bachman, L. & Palmer, A. (2010). Language Assessment in Practice. OUP. Cizek, G.J., & Bunch, M.B. (2007). Standard-setting. A guide to establishing and evaluating performance standards on tests. California: Sage Publications Inc. Council of Europe (2001). The Common European Framework of Reference for Languages. Learning Teaching and Assessment. CUP. Online www.coe.int/lang-CEFR Council of Europe (2009). Manual for Relating Language Examinations to the Common European Framework of Reference for Languages (CEFR). Retrieved from www.coe.int/t/dg4/linguistic/Manuel1_EN.asp Concil of Europe (2009). Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR). Further Material on Maintaining Standards across Languages, Contexts and Administrations by exploiting Teacher Judgment and IRT Scaling. Retrieved from www.coe.int/t/dg4/linguistic/Manuel1_EN.asp Douglas, D. (2010). Understanding Language Testing. London: Hodder. Eckes, T. (2012). Examinee-centered standard setting for large-scale assessments: The prototype group method. Psychological Test and Assessment Modeling, 54(3), 257-283. Figueras, N, North, B, Takala, S, Verhelst, N and Van Avermaet, P (2005). Relating Examinations to the Common European Framework: a Manual. Language Testing 22 (3), 1–19. Harsch, C., Pant, H.A. & Köller, O.(Eds.) (2010): Calibrating Standards-based Assessment Tasks for English as a First Foreign Language. Standard-setting Procedures in Germany. Münster: Waxmann.