Liz Hamp-Lyons

1. Framing research to develop guidelines for developing tests that can be rated according to both scales: The case of writing Liz Hamp-Lyons

2. “Both sets of scales (ACTFL and CEFR) claim to measure the same construct: proficiency”

3. What is proficiency?

4. Let’s focus on one skill and a narrow proficiency range – high level writing

5. What the CEFR knows about assessing high level English writing

6. What the CEFR knows about assessing high level English writing

7. What the CEFR knows about assessing high level English writing Production writing

8. What the ACTFL knows about high level English writing • “It must be noted that the Superior level encompasses levels 3, 4, and 5 of the ILR scale. However, the abilities at the Superior level described in these guidelines are baseline abilities for performance at that level rather than a complete description of the full range of Superior.”

9. What the ILR knows about high level English writing • Writing 5 (Functionally Native Proficiency): • Has writing proficiency equal to that of a well educated native. Without non-native errors of structure, spelling, style or vocabulary can write and edit both formal and informal correspondence, official reports and documents, and professional/ educational articles including writing for special purposes which might include legal, technical, educational, literary and colloquial writing. In addition to being clear, explicit and informative, the writing and the ideas are also imaginative. The writer employs a very wide range of stylistic devices.

10. Wow!

11. ACTFL Superior Level writing (ILR Levels 3 and above) • SUPERIOR • Writers at the Superior level are able to produce most kinds of formal and informal correspondence, complex summaries, precis, reports, and research papers on a variety of practical, social, academic, or professional topics treated both abstractly and concretely. They use a variety of sentence structures, syntax, and vocabulary to direct their writing to specific audiences, and they demonstrate an ability to alter style, tone, and format according to the specific requirements of the discourse. These writers demonstrate a strong awareness of writing for the other and not for the self. • See p.24 of handout booklet for full description.

12. Are the two frameworks intended to be, or indeed claimed to be, equivalent? • They are stylistically different • They are strikingly different in length • ACTFL descriptors are a mix of ‘can-do’s’, personal attributes (“they are able to…”), text characteristics… • CEFR descriptors are superficially ‘can-do’s’ (“Can express him/herself with clarity and precision relating to the addressee flexibly and effectively.”) but in fact are far too vague to be useable as they stand

13. What IS an assessable ’can-do’?’? • Can use effective textual strategies to foreground the principal components of an argument. • Can provide appropriate and salient support for significant issues and main ideas. • Can provide convincing, substantive reasons/justification for a particular position.

14. “The successful establishment of equivalencies would support the validity of both scales.” • OK… • We are not there yet • We may not be on the track to get there • What CAN we do?

15. Framing… to develop guidelines for assessing writing at the higher levels • Guidelines should • Begin from a construct • The construct needs • A theory of second language acquisition • A theory of learning • A theory of written language mastery trajectories • An empirical model of written language use at different levels • An argument (a) language use argument (a) assessment use argument that will weave all these dimensions together appropriately for different audiences/clients

16. Framing… to develop guidelines for assessing writing at the higher levels • Guidelines should • Continue by stipulating the need to obtain perceptions/judgements from a range of stakeholders • Language testing specialists • Teachers of the area under study • Students of the area under study • Score users • Test developers/item writers

17. Framing… to develop guidelines for assessing writing at the higher levels • Guidelines should also • Propose a specification study comparing/ contrasting tasks, input texts, level descriptors across all components of each test (for which the “same construct” is being claimed( • Propose a text linguistic study such as corpus analysis of tasks for difficulty specification; or discourse analysis of what persons scoring specified levels on each test can do within the domain.

18. Framing… to develop guidelines for assessing writing at the higher levels • Guidelines also need to • Stipulate the need for rigorous quantitative studies to test hypotheses deriving from the construct definition and qualitative elicitation stages • Propose consequential validity processes to check the impact of any emerging conclusions on teachers, learners and other stakeholders outside the testing and research enterprises