The tests of sociopragmatic and grammatical awareness in the DESI project: Test constructs and test results

1. The tests of sociopragmatic and grammatical awareness in the DESI project:Test constructs and test results Guenter Nold University of Dortmund guenter.nold@udo.edu

2. Outline • DESI: objectives • Test constructs and a system of task characteristics for language awareness tests • Expert ratings of tasks using system of task characteristics • Item difficulty • Cut-off points for proficiency scales: standard setting • DESI awareness scales • Test results • A link with the CEFR?

3. Objectives of DESI Investigate the causes of the different levels of achievement: cognitive, affective/motivational, social, classroom, home, society Quantitative/qualitative research design Assess Achievement in spoken and written German (L1/L2) and English as FL at grade 9, all types of schools Evaluate English Late Partial Immersion CLIL Implement Changes based on results Objectivesof DESI

4. The English Test Battery • listening comprehension • reading comprehension • reading and text reconstruction (modified C-test) • speaking (computer driven test: Set 10) • semi-creative writing • grammatical awareness • sociopragmatic awareness • (intercultural attitudes) In addition: Bilingualer Sachfachunterricht Englisch/ The German model of late partial immersion(CLIL)

5. Awareness Tests: Test Constructs

6. The sociopragmatic awareness test construct The test construct is focusedbothon facets of areceptive sociopragmatic competencethat EFL learners are supposed to develop in the course of their language studies after four years of instruction in Englishand on theirstrategic abilityto cope with the requirements and restrictions of appropriate language performance. Theirperception and judgement of sociopragmatically appropriate language performanceis partlydue totheir underlying language competenceand partly due to theircompetence to discriminatebetween contextual factors of language use in their L1 (or an additional L2) and in the target language situation. It does not necessarity involve explicit knowledge of sociopragmatic rules and regulations.

7. The sociopragmatic awareness construct in detail More specifically, it is centred on the ability to • perceive and judge situationally appropriate language use, especially with regard to stylistic distinctions and aspects of politeness (e.g. in establishing contact with other people), • recognize speech intentions and to judge their interaction with other speech intentions (e.g. to offer something and to accept an offer), • perceive the coherence of utterances in discourse • and to judge the cohesive markers of texts.

8. Moving from a test construct to a proficiency scale CEFR Curricula Construct Tasks Research Task Characteristics Difficulty Parameters Proficiency Scale

9. 2. The system of task characteristics and expert ratings

10. Task characteristics AaaaaaBb bbbCccccD TC1= Texts with more or less coherent line of thought TC2= Texts with different degrees of textual coherence and cohesion TC3= Perception and judgement of speech functions TC4= Linguistic transfer due to different degrees of prior knowledge / language experience TC5= Different degrees of interpolation needed in solving a task

11. TC 1: Task focuses on texts with more or less coherent line of thought 0: coherence obvious and clear 1: coherence more complex 2: coherence complex, hardly predictable events

12. TC 4: Linguistic transfer due to different degrees of prior knowledge / language experience 0: transfer from L1 (additional L2) possible 1: transfer partly possible 2: interference from L1 (additional L2) likely

13. 3. Item difficulty: Rasch scaling

14. 4. Cut-off points for proficiency scales: standard setting beyond arbitrariness

15. A model to predict item difficulty (multiple linear regression analysis) TC1= Texts with more or less coherent line of thought + TC2= Texts and textual coherence and cohesion + TC3= Perception and judgement of speech functions + TC4= Linguistic transfer, limitations, interference Item difficulty (Dependent Variable)

16. A model to predict item difficulty

17. DESI Sociopragmatic Awareness Levels

18. Task characteristics: M1 M2 M3 M4 M5 M6 0 0 0 0 0 1Kontext: Greeting scene in an office Frank:Good morning, Kate. Kate:Hello, Frank. It’s good to see you. How are you?Frank:  Fantastic! I just got a new job.  I’m O.K. x I’m fine, thanks. And you?  Couldn’t be better.Kate: I’m O.K. See you later. . Sociopragmatic awareness level 1

19. Scale of socipragmatic awareness CN 1 • can perceive and judge appropriate language behaviour, if the connection between speech acts is obvious and the textual means of creating coherence and cohesion are familiar CN 2 can perceive and judge appropriate language behaviour, even if the speech acts and the textual means of creating coherence and cohesion require familiarity with specific sociopragmatic regulations of the target language CN 3 can perceive and judge appropriate language behaviour, even if the sequence of speech acts is more complex. In addition, it is necessary to cope with target language regulations that interfere with L1 regulations. It may also be necessary to be familiar with a wider range of sociopragmatic regulations of the target language.

20. Kompetenzniveaus Bewusstheit Soziopragmatik Result: Sociopragmatic awareness - whole sample

21. Result: Sociopragmatic awareness – different school types Result: Sociopragmatic awareness - Effekt bilingualen Unterrichts

22. The grammatical awareness test construct • The test construct is based on the more or less implicit and partly explicit grammatical competenceof EFL learners in the curricular areas of determiners, tense and aspect, passive, modal verbs, adverbs, word order, dependent clauses such as relative clauses, if-clauses, comparative clauses, infinitive constructions. • It focuses on the strategic competence of learners to locate grammatical errors (internal monitoring) in short written texts and to correct the language items that they have identified as inaccurate. • In terms of a very basic grammatical competence they also fill gaps in sentences by selecting accurate forms from banked items.

23. Task characteristics: grammatical awareness TC1= Semantic function TC2= Morphological-syntactic aspects TC3= Syntactic aspects TC4= Linguistic transfer vs. interference TC5= Time of first presentation of the grammatical feature TC6= Activities: Locating and correcting errors TC7= Grammatical range in item

24. DESI grammatical awareness levels

25. Scale of grammatical awareness CN1 can identify simple morpho-syntactic structures (determiners, relative pronoun) if grammatical options are given. Can correct errors in case the respective form is very basic and familiar. CN2 can locate errors in more complex morpho-syntactic (verb structure, verb in indirect speech) and syntactic structures (word order) and can correct a very small number of these errors. CN3 can correct the errors if the L1 grammatical regulations do not interfere. If there is intereference (e.g. tense and aspect, complementation of the verb), only a very small number of the errors can be corrected. CN4 can correct the errors even if there is interference from L 1.

26. Grammatical awareness level 2

27. Result: Grammatical awareness - whole sample

28. Result: Grammatical awareness – different school types

29. Result of an intervention in the educational system: The German Model of CLIL

30. Ergebnisse zum Einfluss des bilingualen Sachfachunterrichts Results: Intervention – CLIL Late partial immersion program in Germany

31. Thanks! Publications: Nold, G. & Rossa, H. (2006): Sprachbewusstheit. In: Beck, B. & Klieme, E. (Hrsg.) Sprachliche Kompetenzen: Konzepte und Testinstrumente zur Messung der Leistungen im Deutschen und Englischen. Weinheim: Beltz (in press). Eichler, W. & Nold, G. (2006): Sprachbewusstheit im Deutschen und Englischen. In: Beck, B. & Klieme, E. (Hrsg.) Sprachliche Kompetenzen: Konzepte und Testinstrumente zur Messung der Leistungen im Deutschen und Englischen. Weinheim: Beltz (in press). Further DESI publications on the internet 2006 and in Beltz Publishing House in 2006/7 (in preparation).