Developing Rating Instruments for Academic Writing and Speaking Assessment in Austrian University English Departments

1. Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Language Testing in Austria: Towards a Research Agenda Dr Helen Heaney Dr Armin Berger Department of English University of Vienna

2. Chapter outline • Introduction • Background to the project • The design process • Establishing design principles • Componential analysis • Operational analysis and benchmarking • The products of the process • Conclusion Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018

3. Descriptor for Lexico-Grammatical Resources and Fluency C1.2 • Has a good command of a very broad[C2: VOCABULARY RANGE: 112] range of language to express him/herself clearly[C1: GENERAL LINGUISTIC RANGE: 110] and appropriately • Generally maintains a high degree of lexical and grammatical control,as appropriate for the genre, where errors are rare and mostly insignificant[C1: GRAMMATICAL ACCURACY: 114] • Can express him/herself fluently and [...] almost effortlessly [C1: SPOKEN FLUENCY: 129] (i.e. rarely pauses for reasons of grammar or word choice, reducing pauses by using appropriate fillers; reformulations occur mainly for reasons of expressing ideas fully) Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018

4. An expanded model for rating scale development (based on Galaczi et al. 2011)

5. The way forward • Revision and/or fine-tuning of descriptors for writing • More recent research, e.g. Core Inventory for General English (North, Ortega & Sheehan 2011), English Profile • Descriptive scheme of CEFR (CoE 2001), CEFR Companion Volume (CoE 2017) • Systematic performance analysis • Corpus analysis • Post hoc validation of writing scale • Investigating the use of the scales in different contexts • Rater training Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018

6. Bibliography Berger, Armin (2015). Validating analytic rating scales: A multi-method approach to scaling descriptors for academic speaking. Frankfurt am Main: Peter Lang. CoE/Council of Europe (2001). Common European Framework of Reference for Languages, https://rm.coe.int/1680459f97. (02.10.2018) CoE/Council of Europe (2017). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume with New Descriptors. (Provisional Edition). https://rm.coe.int/cefr-companion-volume-with-new-descriptors-2018/1680787989.(02.10.2018) English Profile (n.d.), http://www.englishprofile.org. (02.10.2018) Galaczi, Evelina; ffrench, Angela; Hubbard, Chris; Green, Anthony (2011). “Developing assessment scales for large-scale speaking tests: A multiple-method approach”. Assessment in Education: Principles, Policy & Practice 18(3), 217-237. Heaney, Helen (2013). “Fleshing out CEFR descriptors at C1 and above for the assessment of academic writing in departments of English at Austrian universities”. Dutch Journal of Applied Linguistics 2(1), 43-56. LTC/Language Testing Center (2009). ELTT Mission Statement, https://www.aau.at/en/language-testing-centre/archive/austrian-university-eltt/. (02.10.2018) North, Brian, Ortega, Angeles; Sheehan, Susan (2011). A Core Inventory for General English. London: British Council/EAQUALS, http://englishagenda.britishcouncil.org/sites/default/files/attachments/books-british-council-eaquals-core-inventory.pdf. (02.10.2018) Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018