Quality Management in FL Education

1. Quality Management in FL Education Large Scale Assessment, Procedural Controls and Self Evaluation A critical analysis, with some references to the post-PISA and post-DESI German scene Konrad Schroeder, University of Augsburg

2. Main Points to be addressed • The Traditional System and the Impact of PISA and the CEF • Standards in Large Scale Language Assessment • New Standards in Classroom Assessment: Procedural Controls • The Role of Self-Assessment: Portfolio-Work etc. • Establishing a „New Evaluative Culture“ for Our Schools: Bridging Gaps and Healing Traumas • A Changing Role for National Testing Institutions • Some Golden Rules for Teachers Vilnius Convention May 2011

3. Traditional Modes of Assessment in the FL Classroom • Modes of assessment based on school traditions rather than didactic insight • No proper criteria for the construction of tasks. No construction standards • Subjective tasks, often reproducing the structure of exercises in the course books. Very little task orientation. Very little communicative reliability • Pseudo norm-orientation (→normal probability curve) instead of transparent criteria • Traditional modes of marking and a negative approach based on the counting of errors Vilnius Convention May 2011

4. The Advent of a New Era in FLT (1)PISA and DESI • PISA was targeted at reading comprehension as a basic tool for cultural participation and insight: it was mother-tongue based, but had a decidedly communicative orientation, and its results had a definite bearing on FLT performance. Finland won, but Germany as another wealthy EU country with a well- established educational system, failed: a fact that generated the famous „PISA- Schock“ and a call for a complete change in the system: from “input- orientation“ to “output-orientation“, and consequently for a “new culture of evaluation.” • Internationally speaking, PISA produced two positive results: ▪ Firstly: The call for National Educational Standards, with teaching syllabi and student books based on them. ▪ Secondly: A move towards a monitoring of educational systems based on truly empirical external evaluation (comparative proficiency tests, centralised exams), also with the idea of making it easier to compare national profiles. • DESI later confirmed the PISA results for Germany with regard to English as a FL: Students had substantial deficits in listening and reading comprehension. Vilnius Convention May 2011

5. The Advent of a New Era in FLT (2)The Common European Framework (CEF) The CEF: A tool in many ways revolutionary. Its assets: ▪ It offers a positive approach to language learning by using cando-statements. No counting of errors. ▪ It honours the lower levels of proficiency, giving each of them their own gestalt and thus: right to exist, value. ▪ It emphasises the various facets of oral proficiency, and describes them at the six levels. ▪ It includes scales for a fair number of sub-skills. ▪ The underlying competence model is consensus-based and empirically scaled. ▪ The criteria for the descriptors are taken from the outside world. They have their equivalents in real life. Vilnius Convention May 2011

6. Some Flaws of the Common European Framework • The CEF is unable to scale intercultural competence, which nowadays is a key issue in and an integral part of FLT. • The CEF does not scale language awareness, the development of which nowadays is, again, a key issue in FLT. • The CEF does not scale language learning awareness, the development of which is absolutely necessary in view of European demands and the results of globalisation. • The CEF does not deal with literary understanding (text and media competence). Vilnius Convention May 2011

7. Large Scale Language Assessment as a Cure (1) The Aims Two related objectives: • Comparing the levels of proficiency of groups of learners nationally or internationally, in the context of quality development • Evaluating the level of proficiency of individual learners by way of centralised tests, in order to produce comparable individual results (e.g. meeting an educational or a career requirement, → certificate) Vilnius Convention May 2011

8. Large Scale Language Assessment as a Cure (2) Standards • In the context of quality development: a probabilistic approach. The individual testee is of no interest. • In the context of centralised tests: focus on individual proficiency • Standardised, properly validated tasks (a clear view of the test construct, a clear-cut competence-model, a transparent criterion orientation, an empirical operationalisation through phases of pre-piloting, piloting) • A standardised test administration (sampling of items and students) • An objective scoring and grading (rating scales, benchmarks, rater training) • Empirical development of proficiency scales (Rasch-modeling, standard setting – a process both empirical and based on expertise) • Transparent reporting (generalised results, e.g. in terms of proficiency levels: norm orientation, → normal probability curve) Vilnius Convention May 2011

9. How to establish a „New Culture of Evaluation“ in the FL Classroom (1) Traditional evaluation Subjective tasks Pseudo norm-orientation Error counting Grades Needed in the Classroom: ? External evaluation: Competence models Explicit criterion-orientation Objective rating Standardisation Levelsof proficiency Vilnius Convention May 2011

10. Aims and Standards of Assessment in the Innovative FL Classroom: Aims: • Measuring the achievement of individual learners and, implicitely, the quality of their learning processes over a defined period • Finding out degrees of achievement, and spotting deficits Standards: • An individual approach: the individual testee and their learning processes are in the centre of interest. • Tasks based on the didactic expertise of the teacher (as an expert in the fields of tuition and [process-oriented] testing): criterion orientation • A test administration standardised within the quality level which the individual school can offer • Scoring and grading done within the quality level which the individual school can offer: ad hoc rating scales, ad hoc benchmarks, ad hoc rater training. • Using the CEF as benchmark (as far as language skills are concerned) • Transfering degrees of achievement into marks, following a positive, process-oriented approach Vilnius Convention May 2011

11. Traditional evaluation Subjective tasks Pseudo norm-orientation Error counting Grades Needed in the Classroom: External evaluation: Competence models Explicit criterion-orientation Objective rating Standardisation Levels of Proficiency Innovative tools to diagnose and to measure individual achievement and progress Grades How to establish a „New Culture of Evaluation“ in the FL Classroom (2) Vilnius Convention May 2011

12. The Role of Self-Assessment • Self-assessment in general: a key competence. In the EU the self-assessment of FL competences is an important asset. • The Portfolio format of the Council of Europe: a training-tool for the development of language- and culture-oriented self assessment-skills • Self-assessment in FL course books: Test-Yourself-pages etc. • The psychological truth: If properly trained, learners become well aware of their communicative assets and deficits. Further empirical research is needed in this field. • Schools experimenting in this field (e.g. by including self-assessment into the achievement statements of the school report) report on a dramatic rise in student motivation. Vilnius Convention May 2011

13. Traditional evaluation Subjective tasks Pseudo norm-orientation Error counting Grades Needed in the Classroom: External evaluation: Competence models Explicit criterion-orientation Objective rating Standardisation Levels of Proficiency Innovative tools to diagnose and to measure individual achievement and progress Grades How to establish a „New Culture of Evaluation“ in the FL Classroom (3) How to establish a „New Culture of Evaluation“ in the FL Classroom (3) Self-assessment Vilnius Convention May 2011

14. Traditional evaluation Subjective tasks Pseudo norm-orientation Error counting Grades Needed in the Classroom: External evaluation: Competence models Explicit criterion-orientation Objective rating Standardisation Levels of Proficiency Preparation for, and “making use“ of central exams anddifferent kinds of external evaluation Levels of Proficiency Innovative tools to diagnose and to measure individual achievement and progress Grades How to establish a „New Culture of Evaluation“ in the FL Classroom (4) Self-assessment Vilnius Convention May 2011

15. A Modified Role for National Testing Institutions (1) • The traditional role of national testing institutions has been to develop and administer instruments of external evaluation, basically in the context of quality management and educational standardisation, but also as (semi-) standardised exams for schools (cf. e.g. the British test boards). • These duties will not change, but new responsibilities have recently been added (cf. e.g. the IQB in Berlin): ▪ Writing educational standards (or bringing them up to date) ▪ Developing “learning tasks“ („Lernaufgaben“) as a preparation for central exams and different kinds of external evaluation, thus helping to avoid a narrow-minded “teaching to the test“- approach. Vilnius Convention May 2011

16. A Modified Role for National Testing Institutions (2) ▪ Developing achievement tests based on these tasks, as examples of „good practice“ ▪ Preparing material in the context of advising teachers how to best make use of centralised exams and different kinds of external evaluation. Vilnius Convention May 2011

17. A „New Evaluative Culture“ on the MoveAreas of Action Initiatives at level of individual school:“Assessment and Evaluation” Teacher trainingat university and in-service … … Teaching materialsreflecting assessment needs Proper educational policiesImplementing evaluation systems. Supplying training and materials. Vilnius Convention May 2011

18. And finally: Some Golden Rules (1) As an FL teacher • Be not dismayed: In no other domain of FL teaching has there been such a dramatic change as in student evaluation. Things are changing for the better. Witchcraft is not involved. You can understand the new approach and follow it. • External testing, using validated and standardised tasks and methods, is not negative or educationally dangerous. It is imperative if we take quality management (at group and/or individual level) seriously. • Testing too often (“testeritis“) is demotivating and therefore counter-productive. Traditional testing with its lack of transparence, validity, and reliability is extremely counterproductive. • Do not confound large scale assessment (proficiency testing) with the homemade, process-oriented achievement tests of the FL classroom. The standards of large scale assessment cannot be reproduced on a smaller scale. There is no „large scale assessment light“. Vilnius Convention May 2011

19. And finally: Some Golden Rules (2) • When you write a process-oriented achievement test measuring learning success ▪ have recourse to your didactic and testing expertise. Do not merely follow traditions. ▪ make your criteria transparent. ▪ do not fall victim to a norm-orientation which does not exist: There is no „normal“ distribution of achievement in an FL classroom. ▪ In marking the documents try to follow a positive approach (no error-counting) based on your own, task-specific rating scales. Use the CEF as a reference work only, not as a set of pre- established descriptions. ▪ Develop a clear view of the many domains of FLT that cannot be objectively tested. Vilnius Convention May 2011

20. THANK YOU FOR YOUR ATTENTIONMy thanks go to my former student Dr. Claudia Harsch, Assistant Professor for Language Assessment at the University of Warwick, with whom I collaborated in writing this presentation. Konrad.schroeder@phil.uni-augsburg.de Prof. em. Dr. Konrad Schröder Department of Didactics of English University of Augsburg Germany