Raili Hildén University of Helsinki

1. Raili Hildén University of Helsinki Relating the Finnish School Scale to the CEFR

2. Starting point • “The construction of a comprehensive, transparent and coherent framework for language learning and teaching does not imply the imposition of one single uniform system. On the contrary, the framework should be open and flexible, so that it can be applied, with such adaptations as prove necessary, to particular situations.” (CEFR, 2001, p. 7)

3. How was the Finnish school scale (FSS) designed? • Decision to adapt/adopt the CEFR levels was made by the language experts invited to be members of a national curriculum development working group; approved by the supervising board • Sources consulted: • CEFR – scales for multiple communicative activities • Canadian Benchmarks • Teacher judgement • Working group members commented on the drafts

4. How was the school scale designed? • Several versions were produced based on internal feedback • First round of empirical check for inter- rater consistency (by the team members) • Re-formulation of descriptors with low agreement/consistency • Second round of empirical validation • Agreeing on the current formulations

5. Proficiency levels (and their labels) in Finnish language curriculahttp://www.oph.fi

6. Categories included in the Finnish language curricula Listening comprehension • Themes, text and tasks (1) • Conditions and constraints (2) Speaking • Themes, texts and tasks (monologue and interaction) • Fluency • Pronunciation • Linguistic range • Linguistic control Reading comprehension • Themes, texts and tasks (2) • Conditions and constraints (1) Writing • Themes, texts and tasks (2) • Linguistic range • Linguistic control

7. Research Questions RQ1. What is the level of agreement between judges on the CEFR level of the FSS descriptors? RQ2. How are the Finnish syllabus descriptors related to CEFR levels?

8. Design • FSS level descriptors were split up into 184 statements • Listening 38, Speaking 66, Reading 31 Writing 49 statements • The statements were coded and grouped in terms of communicative activities • Criterion scales used in the rating of FSS descriptors were selected from among relevant CEF scales

9. Design • A sample of 40 Finnish language teaching experts were contacted by an e-mail questionnaire • 20 experts returned the questionnaire • A randomised selection of statements referring to each of the four communicative activities was e-mailed to the raters (in Finnish translation by Huttunen & Jaakkola 2003)

10. Conducting the research Notes: a/ CEFR scale used for the rating task- Overall listening comprehension b/ Anchor descriptors, rated by all raters

12. Conducting the research Notes: a/ CEFR scale - Overall reading comprehension b/ Anchor descriptors, rated by all raters c/ CEFR scale Reading for information and argument

15. RQ1.The range of raters’ agreement Range Note: Ranges of 7 and 8 were checked after the presentation and detected to be due to clerical errors. Thus the ”true” range is From 0 to 6.

16. RQ 1.Agreement with the CEF levelsExample of a descriptor with a low level of agreement between raters.“Can identify the writer’s bias and the purpose of the text and locate and integrate several specific pieces of information in a longer text. Can quickly identify the content and relevance of new items deciding whether closer study is worthwhile” Level variance=2,37 FINSS=B21 CEF=6 CEF reference B2

17. RQ 1.Agreement with the CEF levels Example of a descriptor with a high level of agreement between raters. “Can write the alphabet of the language and all numbers and numerals. Can write down basic personal identification information and write a small number of familiar words and simple phrases.” variance=0 FINSS=A11 CEF=1 CEF reference A1

18. RQ2.Agreement between syllabus level descriptors (syllcode) and the original CEFR levels (levcode) Note: A quite good level of agreement was oberved (65%). There is, however, some tendency for an overestimation: 19 descriptors (10%) were assigned to a lower level while 46 (25%) were assigned to higher level.

19. RQ1. Range distribution per skill The average range is 2.66 for the whole pool. The range distribution per skill can be seen in the boxplot above, showing no clear differences between the four skills.

20. RQ2. Absolute agreement per skill between syllabus level and CEF level

21. RQ1. Factors affecting the ratings (= raters are quite ”homogeneous”)

22. RQ2. Agreement between individual rating and original (initial) levels (Syllabus - syll & CEF - level)

23. Plans for further exploration • Calibrating the FSS descriptors • Exploring the link to the Canadian Benchmarks in more detail • Re-formulating or removing problematic descriptors • Empirical validation and exemplication of the FSS scales through benchmarks for comprehension tasks and for oral and written performance samples

24. Summary • The correspondence between the new Finnish school scale (FSS) and the CEFR scales were studied. • Research question 1: What is the level of agreement between judges on the CEFR level of the FSS descriptors? • Research question 2: How are the Finnish syllabus descriptors related to CEFR levels? • 20 experienced raters judged FSS descriptors using relevant CEFR scales • A good agreement was reached: 65% of the FSS descriptors were assigned to the original CEFR levels. For the rest of the cdescriptorts, some tendency of overestimation was observed. • Inter-rater agreement was also quite good.

25. Acknowledgement I wish to thank • Sauli Takala, University of Jyväskylä • Feljanka Kaftandieva, University of Sofia for their help in carrying out this study.