THE DUTCH “PAMPHLET” AND THE USE OF THE CEFR

1. THE DUTCH “PAMPHLET” AND THE USE OF THE CEFR UNIVERSITY OF LEUVEN Piet Van Avermaet & Veerle Depauw Kransjka Gora, May 2004

2. OVERVIEW • The Pamphlet of the Dutch Language Union • Misuse and impact of the CEFR? • The CEFR as a point of reference or as a prescriptive tool? • Example of how VDAB uses the CEFR • The CEFR providing a meta language for communication

3. The Pamphlet of the Dutch Language Union 1 Objectives for testing have to be set socially and educationally. 2 These objectives for testing can subsequently be related to regional (Flemish), national (Dutch) or international (e.g. CEFR) frameworks or collections of objectives. Regional or national frameworks are preferably related to the Common European Framework of Reference. 3 The Common European Framework of Reference should as such be used as a frame of reference and not as a prescriptive instrument. 4 Linguistic competence should be tested functionally in a user context. 5 When describing and assessing linguistic competence, differentiation with regards to content is needed; differences as to social, educational and professional roles (contexts of language use) will have to be visible in tests and in language products to be assessed.

4. Why this pamphlet? Concern that CEFR would become more and more a prescriptive instrument? Based observation, communication Is concern legitimate/justifiable? A few examples

5. Misuse of the CEFR? • CEFR a non prescriptive instrument misuse • Do we all see the CEFR as a point of reference instead of a normative instrument? • Conflicting discourses or mismatch between discourse and behaviour discoursebehaviour point of reference …a basis for… not prescriptive item bank compendium templates • What is a template? A special shape or pattern used to make identical ...

6. Mismatch between discourse and behaviour? How comes that there is this mismatch? • We want to remain faithful to basic principles and goals of CEFR • But we also assume/claim that everybody is familiar with the CEFR, uses it as a basis for test development, syllabus design, etc… I.e. use as a rather normative instrument • However, who are the people who use it as such? The same who state that the CEFR is a point of reference?

7. The CEFR and rationalisations What about the others? The not us, policy makers, teachers, curriculum designers, language planners, etc… Why do they use it? Why have they decided to use the scales? Rationalisations? The attraction of the dominant discourse, of power subversive mechanism

8. IMPACT of CEFR We don’t now! What is the actual impact (political, educational, societal)? What is current behaviour in Europe in relation to CEFR? What are people’s perceptions of the CEFR? How do these perceptions relate to their behaviour? What made people decide to use it? How do people use it? How do they interpret it? How would they use if they knew it in depth?

9. CEFR as prescriptive instrument To what extent is it acceptable, desirable to impose a norm? Do we want to change the CEFR from a point of reference to a more prescriptive instrument? But we don’t want to! We want to remain faithful to its basic principles Wait, let’s have a look at a few things that are going on now in Europe: idea of European items (decontextualised?, across languages?) specifications across languages one construct underlying the CEFR templates for development of test items CEFR scales as basis for and an item bank for Barcelona indicators project

10. How do I know that my B1 is your B1? • How do we have to interpret this question in relation to what I just said? • How do I know that my mojito is your mojito? • Different rums how much rum • lemon or lime how many drops of lime/lemon • syrup of sugar cane, real sugar cane, white sugar, vanilla sugar • different mints how many leaves • what is interaction effect of these different ingredients on a mojito? • Do all ingredients of a mojito have to be exactly similar to call it a mojito? Can they differ according to the ingredients available, to the local context?

11. How do I know that my B1 is your B1? Suppose you claim your exam is at B1 fair question What is rationale behind having an exam at B1? impact of CEFR? Who wants to know what people have to do at B1? Who wants to know whether a person is at B1? Or do we want to know what people have to do with language in a given situation? Suppose you start from a needs analysis and develop real life tasks: different features are at different levels. What claim should one make? How can one answer the question how do I know that my B1 is your B1? Do all items have to be at B1? Do all testtasks have to be at B1? Do all components of an examination have to be at B1?

12. How do I know that my B1 is your B1? • Can we answer the B1 question for examination/items/testtasks from a content and consequential perspective, from a functional perspective? • context validity features (Weir 2004) • for each feature specifications along the CEFR scales. Is this possible? • What is effect of interaction of features on level? • What if an item/test task consists of different levels for different goals and/or features? B1 A2 1A2 A1 PROFILING B1 B1 2B2 A2

13. How do I know that my B1 is your B1? Answering this question is even more problematic if CEFR is basis for test development. In that case danger is that we end up with items/test tasks that are perfect at a level, but have low real life validity. A kind of artificial test tasks based on an approach of development through segmentation. In real life situations people often have to perform in a more integrated way often with features at different levels. How far can a test be from real life?

14. Who are we? • VDAB: one of the main providers of vocational training for adults in Flanders • An important challenge: we meet a lot of non-native adults (DSL) who want to attend a vocational training course • The development of a (language) curriculum with (language) assessment procedures at important stages in the learning process (vocational & language)

15. The language curriculum in Flanders BASIC LEVEL (richtniveau 1.1) Provided by another supplier

16. CEFR NATIONAL FRAMEWORK SHORT DESCRIPTION RECOMMENDED LANGUAGE TRAINING DURATION C2 In total: 1200 hours 15 hours a week = one and a half year! C1 Richtniveau 4 240 h B2 Richtniveau 3 240 h B1 Richtniveau 2 480 h A2 Richtniveau 1.2. 1.1. + 1.2. : 240 h A1 Richtniveau 1.1. (=basic proficiency)

17. A few important characteristics of our language learners • They are aiming at a job, as soon as possible • They have a financial need to find a suitable job • Language learning is not a goal as such, but rather a means

18. No mastery level before entering the vocational courses • Due to the learners’ profile • Due to our view on language learning: performing tasks in a natural environment instead of exclusively learning the language in the classroom

19. NEEDS ANALYSIS A SET OF SPECIFIC GOALS LANGUAGE CURRICULUM & ASSESSMENT

20. The language curriculum in Flanders (a professional perspective ) TRAINING FLOOR VDAB-trainee Example: assessment ‘The Springboard’ SPECIFIC LANGUAGE VDAB-trainee Example: the Elevator Provided by another supplier BASIC LEVEL

21. A sample of the global scale B2 Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation … B1 Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken.

22. The CEFR is a very useful instrument A meta language for communication A point of reference for comparing and profiling But above all a compendium for more than only testing and assessment Some, however, have doubts and say that the emperor has no clothes. However, in the light of the above (ie the danger of becoming more and more prescriptive) does the emperor really need clothes? Or maybe he needs different sets of clothes for different occasions. The same question for the need of an underlying construct. Does the CEFR need one underlying construct? What if people start from another construct? What we need in the first place is a clear rationale and clear test specifications for different examination contexts (for each test) which can be compared to each other using the CEFR as a point of reference and can be used for profiling tests.

23. The CEFR is a very useful instrument • We much ask ourselves how far we want to go in using the CEFR as a prescriptive instrument and in the development of instruments along this line. • How far does Europe want to go? • How far do national education boards want to go? • How far do schools, teachers, employers, students, candidates want to go?

24. The CEFR is a very useful instrument Tests and frameworks are powerful tools and have enormous impact. Instruments like templates, itembanks, specifications etc, increase their power. A European item bank looks great. And might be ok (although: decontextualised items across languages) as long as it functions as a point of reference like the CEFR. But we know that the CEFR is becoming more and more used as a prescription. We don’t have to make our own mojito anymore. We buy it (mixed, shaken and stirred). The best. Quality ascertained. And reliable: the same for everybody and everywhere HOWEVER

25. Languages and people =coca cola