Comparing the incomparable? Assessing speaking and writing Part Two

1. Comparing the incomparable?Assessing speaking and writingPart Two Ülle Türk University of Tartu Estonian Defence Forces 23rd CSW, Tampere, 27-29 March 2009

2. Questions • Which aspects should we focus on when we assess students’ productions? • Which is more effective: holistic or analytic assessment? • How can we standardize our assessments? • How can we involve students in the process of assessing their speaking and writing skills?

3. Remarks on CEFR • Recommendation of the Committe of Ministers to member states on the use of C of E’s CEFR and the promotion of plurilingualism (CM/Rec (2008) 7) • Explanatory note • The CEFR is purely descriptive – not prescriptive, nor normative; • The CEFR is language neutral – it needs to be applied and interpreted appropriately with regard to specific languages • The CEFR is context neutral – it needs to be applied and interpreted with regard to each specific educational context in accordance with the needs and priorities specific to that context • ......

4. Brian North, Eurocentres • NOT a harmonisation tool • “We have NOT set out to tell practitioners what to do or how to do it.We are raising questions not answering them.” • NOT a theory of language or skills development • Scales describe learning outcomes, learner behaviours, notthe invisible processes involved. • NOT a test specification • Scales and lists can be consulted when drawing up a taskspecification (Ch4) or defining assessment criteria (Ch5) butneed reference to detailed specs for language & context

5. Development of descriptors • Intuitive Phase: • Creating a pool of classified, edited descriptors • Qualitative Phase: • Analysis of teachers discussing proficiency • 32 teacher workshops sorting descriptors • Quantitative Phase: • Teacher assessment of 2800 learners on descriptorchecklists(500 learners, 300 teachers) • Teacher assessment of videos of some learners • Interpretation Phase: • Setting “cut-points” for common reference levels

6. Ambilingual Proficiency Comprehensive Operational Proficiency Adequate / Effective Operational Proficiency Limited Operational Proficiency Basic Operational Proficiency Survival Proficiency Formulaic Proficiency EWENS: Well-educated Native Speaker D2Genuine bilinguals (+ Beckett etc.) D1Language professionals: Interpreters,translators, some university professors C2Highly successful learners C1 B2 B1 A2 A1 “Life beyond C2”

7. Salient Characteristics D? • Apparent ambilingualism: • Convey, elaborate or translate to explicit expression thenuances and subtleties of their own and of others’meaning by exploiting a comprehensive knowledge of thelanguage to do so • function in all situations to all intents and purposes exactlyas the mother tongue; use the language in asophisticated, natural, accurate manner apparentlyindistinguishable from the performance of a nativespeaker

8. CEFR Levels: Key Problems • Danger of differing interpretations for different languages • Under-definition of C2, + some reversals of C1/C2descriptors (ALTE, DIALANG, Catalonia) • Weak definition of socio-linguistic competence (andsome contradiction to Cambridge qualitative research) • Unrealistic expectations in relation to receptive skills

9. Under-definition of C2 • Mostly uncalibrated as very few C2 descriptors calibratedin CEFR/Swiss project • Integrate suitable descriptors from ALTE,DIALANG, Catalonia, Portfolio bank • Occasional C1/C2 reversals • Investigate cases; Incorporate insights fromqualitative analysis of samples (e.g. Cambridge) • C1 descriptors tend to be more concrete, C2 descriptorsless so – but try to avoid “native speaker” attributes • Define Level D, at least in outline, to give upperboundary; Consult curriculum descriptors

10. Reliability of assessment • Do all markers agree on the mark I got? • = inter-marker reliability • If the same marker marks my test paper againtomorrow, will I get the same result? • = intra-marker reliability • Objectively marked tasks • Subjectively marked tasks

11. Methods of marking • Analytical marking • according to detailed criteria • The final score is a composite of all the subscores or a profile. • Holistic or impression marking • There is no breakdown intoseparate marks for separate aspects of writing skill, but some criteria still haveto be kept in mind.

12. Holistic marking scheme

13. Sara Cushing Weigle, 2002: 121

14. Scale functions (CEFR, p 37-38) • user-oriented scales • report typical or likely behaviours of learners at any given level • what the learner can do • tend to be positively worded, even at low levels • often holistic, offering one descriptor per level • assessor-oriented scales • guide the rating process • how well the learner performs • often negatively worded even at highlevels • Some are holistic scales, others are analytic scales.

15. Task specific vs general scales • Task specific easier to use, but more time-consuming and expensive • Designing different scales • Training assessors to use them • A combination of the two: • Linguistic competence the same (Ch 5) • Task-specific aspects different (Ch 4)

16. Addressing audiences C1Can give a clear, well-structured presentation of a complex subject, expanding and supporting points ofview at some length with subsidiary points, reasons and relevant examples. • Can handle interjections well, responding spontaneously and almost effortlessly. B2+ Can give a clear, systematically developed presentation, with highlighting of significant points, andrelevant supporting detail. • Can depart spontaneously from a prepared text and follow up interesting points raised by members ofthe audience, often showing remarkable fluency and ease of expression. B2Can give a clear, prepared presentation, giving reasons in support of or against a particular point ofview and giving the advantages and disadvantages of various options. • Can take a series of follow up questions with a degree of fluency and spontaneity which poses no strainfor either him/herself or the audience.

17. Reading for information and argument C1Can understand in detail a wide range of lengthy, complex texts likely to be encountered in social,professional or academic life, identifying finer points of detail including attitudes and implied as well asstated opinions. B2+Can obtain information, ideas and opinions from highly specialised sources within his/her field. • Can understand specialised articles outside his/her field, provided he/she can use a dictionaryoccasionally to confirm his/her interpretation of terminology. B2Can understand articles and reports concerned with contemporary problems in which the writers adoptparticular stances or viewpoints.

18. How many levels? • Proficiency levels too broad to trace achievement • The higher the level the longer it takes to reach it • Split into narrower levels • Finnish National Curriculum levels • NB! There is a level between B2 and C1!

19. Examiner training • NB! The less analytic and mechanical the method of marking, the morehighly skilled and trained examiners need to be. • Standard setting • Standardisation • Monitoring examiners • Evaluation of examiners

20. Questions to ask • What competences should my students have in • Spoken interaction • Spoken production • Written interaction • Written production • Mediation? • What tasks/activities should they be able to perform to demonstrate their mastery of the competences? • How well should they be able to perform them? • How are we going to assess their written/spoken performance reliably?

21. Sources • Council of Europe: Language Policy Division: http://www.coe.int/T/DG4/Linguistic/Default_en.asp • European Language Portfolio: www.coe.int/portfolio/ • Cushing Weigle, Sara (2002) Assessing Writing, Cambridge University Press, Cambridge.