Standardisation of judgements: reading and writing

1. Standardisation of judgements: reading and writing Ülle Türk University of Tartu/Estonian Defence Forces ulle.turk@ut.ee/ylle.tyrk@mil.ee EUROPOSSĄJUNGA

2. Reading EUROPOSSĄJUNGA

3. Training with standardised samples for reading • Phase 1: Illustration – a first assessment of the level of one text and its corresponding tasks and items • At what CEFR level can a test taker already answer the following item correctly? • Phase 2: Controlled practice – different texts with their corresponding tasks and items are assessed under the guidance of the co-ordinator • Phase 3: Individual assessment of the remaining items in chunks of 4-6 items The session continues until a spread of results of one and a half levels is achieved. EUROPOSSĄJUNGA

4. Benchmarking local samples of reading • Individual rating followed by (pair) discussion leading to group consensus • Discussion of spread across individual ratingsand iteration until suitable agreement (spread equal to one and a half levels) is reached Ratings of the CEFR standardised samples should be analysed statistically in order to • Confirm the relationship to the levels and • Calculate intra-rater reliability (consistency) and inter-rater reliability EUROPOSSĄJUNGA

5. Task 4: Items 1–4 • Text Characteristics: • Text source: magazine • Authenticity: authentic • Discourse type: mainly argumentative • Domain: public • Topic: other/politics • Nature of content: fairly extensive abstract content • Text length 350 words • Vocabulary: extended vocabulary • Grammar: wide range of complex structures • Text likely to be comprehensible by learner at CEF level: C1 EUROPOSSĄJUNGA

6. Item 1 Item type: Selected response/multiple choice Operations: Infer implicit main idea/gist Item level estimated: A2/B1 Key: B Per cent correct: 90.5% Discrimination (item/total correlation): .31 Item 2 Item type: Selected response/multiple choice Operations: Make explicit inferences about the writer’s mood Item level estimated: C2 Key: B Per cent correct: 36.1 % Discrimination (item/total correlation): .45 Items 1 & 2 EUROPOSSĄJUNGA

7. Item 3 Item type: Selected response/multiple choice Operations: Make inferences about implicit main idea/gist Item level estimated: C2 Key: B Per cent correct: 43.3 % Discrimination (item/total correlation): .20 Item 4 Item type: Selected response/multiple choice Operations: Make inferences about implicit detail Item level estimated: B1 Key: C Per cent correct: 83.6 % Discrimination (item/total correlation): .43 Items 3 & 4 EUROPOSSĄJUNGA

8. Task 5: Items 5–8 • Text Characteristics: • Text source: magazine • Authenticity: authentic • Discourse type: mainly expository • Domain: public/personal • Topic: other/contemporary life • Nature of content: mainly abstract content • Text length 418 words • Vocabulary: extended vocabulary • Grammar: mainly complex structures • Text likely to be comprehensible by learner at CEF level: C1 EUROPOSSĄJUNGA

9. Item 5 Item type: Selected response/multiple choice Operations: Infer implicit detail Item level estimated: B2 Key: B Per cent correct: 65,9 % Discrimination (item/total correlation): .52 Item 6 Item type: Selected response/multiple choice Operations: Evaluate implicit opinion Item level estimated: C2 Key: C Per cent correct: 49,2 % Discrimination (item/total correlation): .32 Items 5 & 6 EUROPOSSĄJUNGA

10. Item 7 Item type: Selected response/multiple choice Operations: Make inferences about implicit main idea/gist Item level estimated: C1 Key: C Per cent correct: 52,1 % Discrimination (item/total correlation): .36 Item 8 Item type: Selected response/multiple choice Operations: Evaluate implicit attitude/conclusion Item level estimated: B2 Key: A Per cent correct: 74,4% Discrimination (item/total correlation): .24 Items 7 & 8 EUROPOSSĄJUNGA

11. Writing EUROPOSSĄJUNGA

12. Training with standardised samples for writing • Phase 1: illustration with about three standardised performances – 60 minutes • Phase 2: controlled practice with 3–5 standardised performances – 60 minutes • Phase 3: individual assessment of 3–5 standardised performances – 60 minutes The session ends when the vast majority ofparticipants agree on the level, with the spread not exceeding one and a half levels. EUROPOSSĄJUNGA

13. Benchmarking local samples of writing • Individual rating and group discussion of high, middle and low performances – 60 minutes • Individual rating of about five more performances – 60 minutes EUROPOSSĄJUNGA

14. Sample task 1 • Your English Teacher has asked you to write a story. • Your story must have the following title: The most important day of my life • Write your story on your answer sheet. EUROPOSSĄJUNGA

15. Example answer 1 The most important day of my life. During a lifetime there is so many days you could call,the mostimportant day of your life”. It could be theday you chose wich school you are going to, or what youwant to work with the rest of your life. Anotherimportant day is when you get married, or you chosewhere to live. But most of all it must be a veryimportant day when you give birth to a child. I thinkthat changes everything you have been doing until then.Than you have to realice that somebody are moreimportant than yourself. EUROPOSSĄJUNGA

16. Sample task 2 This is an extract from a letter which you receive from theWorld OpinionOrganisation. We are carrying out an international survey onfamilies around the world and the importance offamily relationships. Please write us a report forour survey outlining the typical family situationin your own country and suggesting how you thinkthe situation might change in the future. Write your report.Write approximately 250 words. EUROPOSSĄJUNGA

17. Sample task 3 You have read the extract below as part of a newspaper article on the loss ofnational and cultural identity. Readers were asked to send in their opinions. Youdecide to write a letter responding to the points raised and expressing your ownviews. ‘We are losing our national and cultural identity.Because of recent advances in technology and theeasy availability and speed of air travel,different countries are communicating more oftenand are therefore becoming more and more alike.The same shopping malls and fast food outlets canbe found almost everywhere. So can the same typesof office blocks, motorways, TV programmes andeven lifestyles. How can we maintain thetraditions that make each nation unique?’ Write your letter. Do not write any postal addresses. EUROPOSSĄJUNGA

18. Sample task 3 • You recently entered a competition and have just received this letter from theorganiser. Read the letter, on which you have made some notes. Then, using all theinformation in your notes, write a suitable reply. • Write a letter of between 120 and 180 words in an appropriate style on the oppositepage. Do not write any postal addresses. EUROPOSSĄJUNGA

19. EUROPOSSĄJUNGA

20. Sources • Manual for relating language examinations to the Common European Framework of Reference for Languages – a preliminary pilot version: http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html • The CEFR Grid for Writing, v. 2.: • The Final Grid for analysing reading and listening items: www.ling.lancs.ac.uk/cefgrid EUROPOSSĄJUNGA