Use of qualitative methods in relating exams to the Common European Framework: What can we learn?

1. Use of qualitative methods in relating exams to the Common European Framework: What can we learn? Spiros Papageorgiou Lancaster University The Third Annual Conference of EALTA Krakow, Poland, 19 – 21 May, 2006

2. EALTA Guidelines for Good Practice in Language Testing and Assessment LINKAGE TO THE COMMON EUROPEAN FRAMEWORK • What evidence is there of the quality of the process followed to link tests and examinations to the Common European Framework? • Have the procedures recommended in the Manual and the Reference Supplement been applied appropriately? • Is there a publicly available report on the linking process?

3. Overview • Linking exams to the CEFR • Use of qualitative methods • Verbal protocol analysis • Small group research • Examples from two studies • Conclusion: From theory to practice

4. Linking exams to the CEFR • A process involving judgements by a panel • Familiarisation • Specification • Standardisation

5. The Judges’ role in linking exams to the CEFR CEFR Judges Linking claim Manual Test

6. Using qualitative methods: Aims • Judges can provide insights into: • Decision-making • The CEFR scales • The linking process • How important is it to gain insights in all these? • Validity of the linking claim

7. Study 1: Verbal Protocol Analysis-Lancaster University • Familiarisation task from the Manual • Framework: Ericsson and Simon (1993) • Design: Green (1998), Banerjee (2004) • 6 informants • 3 sets of descriptors (Kaftandjieva and Takala, 2002) from Table 2, CEFR • 2 task layouts

8. Issues for the design of the study • Training • Language • Kind of verbal reports • Sample size • Length • Coding • Task • Procedure

9. Terminology problems/lack of definition (1) • R5 I can understand texts that consist mainly of job-related language. (B1) • “understand is such a vague term!” (I4)

10. Terminology problems/lack of definition (2) • W6 I can write short simple messages relating to matters in areas of immediate need. (B1) • “How short is short?” (I5)

11. Descriptor units-stand alone nature (1) • R2 I can read articles concerned with contemporary problems in which the writers adopt particular stances or viewpoints. (B2) • R13 I can read reports concerned with contemporary problems in which the writers adopt particular stances or viewpoints. (B2) • “to me an article is an easier text type to read rather than a report.” (I1)

12. Descriptor units-stand alone nature (2) • W11 I can describe impressions. (B1) • W15 I can describe experiences. (B1) • “experiences is more than impressions. It’s got feelings and images. So, because impressions is only feelings and also the description is difficult, so that’s why I gave it B2 and not B1”. (I2)

13. Key words • L10 I can understand films without too much effort. (C1) • “without too much effort. So, it comes natural, I don’t have to struggle. So, I thought it is C1.” (I3) • “that is the sort of thing a B1 person should be able to do. The key phrase I think that helped me was without too much effort.” (I6)

14. Inconsistencies • “TV programmes”: in B1 and B2 Listening only • “this […] talks about TV news programmes when there’s already one talking about TV news programmes in B1 so it has to go somewhere else. Whether it should have gone the other side of B1 whether it is even lower I am not sure”. (I6)

15. Summary of findings-VPA • Interaction with the scales • Decision making • Problems with wording of descriptors • Validity of the linking claim during Familiarisation

16. Study 2: Small Group Research-London • Specification of the Trinity College London GESE and ISE suites • 10 participants • Framework: Small group research (Davidson and Lynch, 2002) • Specification Forms of the Manual • Recordings of group work • 3.5 hours of discussion fully transcribed • Atlas.ti • Aim: validity of Specification claim

17. Research design issues • Choice of participants • How to group them • Roles: what the individuals do and how they communicate within the group • Cohesiveness: fitting together of group members • How can these affect the validity of the linking claim? • What other factors might exist?

18. Fitness of descriptors • Descriptors do not always match the test content

19. Prior knowledge/Bias of insiders • Judges likely to see the test through claims already made • Decision making: confirmatory rather than exploratory

20. Predicting behaviour of learners • Major factor in decision making: predicting what test takers can do

21. Test specifications • Test specifications: very influential

22. CEFR and Manual: problems during Specification • Terminology problems/lack of definition • Tasks and communicative tasks • Evidence vs. inferences • Real life vs. exam environment • Young learners • Not everything is scaled • Purposes and functions

23. Summary of findings-SGR • The Specification process: not without problems • Group dynamics • Validity of the linking claim during Specification • Transparency

24. Conclusion: From theory to practice • Linking claim and decision making: Validity, consequences, ethics • Qualitative & quantitative analysis of judgements: complementary • Learn more about the use of the Framework: linking exams and transparency • Validation of proficiency scales: insights from actual use

25. THANK YOU!!! Contact details Spiros Papageorgiou Dept of Linguistics & English Language Lancaster University LA1 4YT UK s.papageorgiou@lancaster.ac.uk