Users’ reactions to innovative computerised feedback – the case of DIALANG

1. Users’ reactions to innovative computerised feedback – the case of DIALANG Ari Huhta Centre for Applied Language Studies University of Jyväskylä Finland

2. What is DIALANG? • computerized diagnostic language assessment system in 14 languages (www.dialang.org) • feedback (& activities) that help learners to • diagnose their strengths and weaknesses • plan further language learning • raise awareness • unusually wide range of feedback for a test; partly unique / innovative, e.g. self-assessment

3. What is ’feedback’? Why feedback? In educational contexts FB can be: • ”… any communication or procedure to inform a learner of the accuracy of a response, usually to an instructional question” (Knowledge of the Correct Response) • but also ”… other information such as precision, timeliness, learning guidance, motivational messages, ... critical comparisons, and learning focus” • ”… any message or display that a computer presents to the learner after a response” (Mory 2004) • improve performance & learning (directly) • but also to motivate, make learning more interesting, encourage different learning, raise awareness, …

4. Why study (DIALANG) feedback? • feedback is an integral part of all behaviour, performance and learning • social psychology, organizational theory, education ... • feedback from tests is not much researched (cf. teachers as assessors) • feedback from computerized (language) tests is particularly little researched

5. DIALANG FEEDBACK

6. Research topics / questions This study: • What are users’ reactions to different types of feedback in DIALANG? How do new types of feedback compare with more traditional feedback? • Are users’ reactions associated with their background characteristics (e.g. sex, age, language use, language proficiency)? How might possible associations be explained? -------------------------------------------------------- • related, more in-depth studies users’ reactions to self-assessment and Vocabulary Size Placement Test

7. Research methods & data • Quantitative & qualitative (cross-sectional) • Questionnaire • 553 respondents in two countries • mostly non-language majors at universities (typical users) • Interviews of users and their teachers • Written reports from users & teachers

8. Users’ ’reactions’? Defined in terms of • 1) Reading / using particular feedback? • 2) Interest in particular feedback? • 3) Perceived usefulness of particular feedback? • 4) Other reactions (e.g. ease of use) • Likert scale + open-ended questions • cf. Technology Acceptance Model

9. Who were the respondents?

10. Age of respondents

11. # of languages & tests taken

12. Most common test languages

13. Results: Overall reaction to FB- use & interest & usefulness

14. Reactions to Level (test result)

15. Reactions to Level (test result) • the most popular part of feedback • the fact that the meaning of A1 etc was defined was appreciated • associations with user background: • correlated with proficiency: higher proficiency – more positive reactions • men were less interested in this than women

16. Reactions to Vocabulary Size Placement Test feedback / result

17. Reactions to Vocabular Size Placement Test & its feedback • VSPT and its result was the most controversial part of DIALANG although, overall, the reactions were positive • strong affective reactions, opinions rather divided • “Interesting, I had never thought of a way like this to test language proficiency. Fun to do!” • “Absolute nonsense! This can’t be my level!” • background: in general, those who got low VSPT results regarded it very negatively, whereas those who got good results regarded it very positively • in particular, women at level C1/C2 liked VSPT feedback • explanation: VSPT scoring is affected by test-taking strategy and produces very low scores too easily

19. Reactions to immediate item review • almost 60% preferred immediate review, 30% post-test review, 10% preference • Why like: want to know immediately, remember better if see the result right away, want to learn (during test) • Why don’t like: disturbs test-taking, can be depressing • background: • beginners liked it more than advanced learners • in particular, C-level (C2) learners considered it less useful and interesting • C-level men used it particularly infrequently • possible explanation: • many beginners & intermediate learners want to learn from tests?? • advanced learners make so few mistakes that they may feel immediate feedback is waste of time??

20. Reactions to post-test item review • less popular than immediate review but a fair number of users absolutely want to have it • Background: • more experienced users (# of languages used & length of language studies) considered this more interesting & useful • possible explanation (very tentative): • more experienced language learners may find information about subskills interesting??

21. Reactions to self-assessment feedback (comparison of SA and test result)

22. Reactions to self-assessment feedback • according to users, SA is important part of (a system like) DIALANG, • as ‘popular’ as e.g. item review (joint second) • background: • men had more often very negative opinions compared with women • the youngest group (under 18) found it less interesting than the older groups • possible explanation: • younger learners may not (yet) be so interested in the kind of reflection on one’s skills involved in self-assessment??

23. Reactions to self-assm. information (why SA and test result may not match)

24. Reactions to SA information • the least used / interesting / useful part of feedback • Background: negative reactions were associated with • higher proficiency, especially C1/C2 women • more experience in lg learning (more lgs studied) • male respondents (more very negative opinions) • young age (under 18) • positive reactions were associated with: • lower proficiency, especially A1/A2 men • less experience in lg learning (fewer lgs studied) • men were divided in their opinions of SA information • possible explanations: • if SA and test agreed, no need to read this • beginners have less experience in what is described in SA information and may find it new & potentially useful; advanced learners may feel they already know all that

25. Reactions to extended descriptions

26. Reactions to extended descriptions • fairly interesting, useful and used but less so than e.g. item review • background: • men made more often very negative evaluations of this feedback than women

27. Reactions to advice

28. Reactions to advice • quite popular but not among all users – surprise? • Background: • high proficiency (C1/C2) & more experience  more negative views of advice • men had more very negative views than women • A1 / A2 learners were divided: they often chose it as one of the 3 most useful types of FB but many also chose it as one of the least useful types of FB • possible explanation: • C-level learners no longer need advice, whereas A-level learners do, but some of them may feel the particular advice in DIALANG is not useful??

29. Implications / questions • implications for DIALANG and/or other such computerized systems • users clearly like to have feedback about their self-assessment • younger learners may need to be motivated first • good to have two types of item review: immediate and post-test, based on choice • preferences differ; level of proficiency

30. Implications / questions • VSPT divides opinions: love – hate • scoring needs revising • advice appears useful for beginners in particular • challenges ”I don’t bother to read the advice because I know I’ll do things my way anyway” ”... you don’t remember them any more when you study” • longitudinal studies • how best integrate systems like DIALANG into teaching • (long-term) effectiveness of feedback?