Motivating students to engage with assessment and self-regulate their own learning

1. Motivating students to engage with assessment and self-regulate their own learning Ali Wyllie and Ingrid Nix Lecturers in Learning and Teaching Faculty of Health and Social Care HEA e-learning for health Conference –S3BP6 (17 July 2009)

2. Research context • Competing priorities of work and study mean HSC students selectively engage in computer-based assessment • HSC students typically low in confidence in ICT • How can we: • increase their motivation • enhance self-regulation (ability to manage own learning).

3. Learning with Interactive Assessment (LINA) project Our research questions: • What features would enhance learner’s self-regulation and motivation to engage? • Would a confidence indicator tool and a learning log promote a more reflective approach to learning and formative assessment?

4. LINA research areas Does variety of media make Qs more motivating (e.g. audio, animations)? Does this encourage reflection and evidence-based practice? Does narrative within sequences make a difference? Does choice make a difference, selecting easier or more difficult (e.g. to make more effective use of study time)? Does this encourage reflection on choices made & input of additional thoughts? In what ways do students draw on Feedback?

5. LINA website Animation window

6. Re-engineering Assessment Practices ‘Good feedback practice is anything that might strengthen the students’ capacity to self-regulate their own performance’ Nicol and Macfarlane-Dick (2006:205) REAP identifies 7 principles of good feedback practice,as feedback which: 1. helps clarify what good performance is (goals, criteria, expected standards) 2. facilitates the development of self-assessment (reflection) in learning 3. delivers high quality information to students about their learning 4. encourages teacher and peer dialogue around learning 5. encourages positive motivational beliefs and self-esteem 6. provides opportunities to close the gap between current and desired performance 7. provides information to teachers that can be used to help shape teaching.

7. Method – research instruments 1. LINA website introducing two topics of 10 questions. 2. Feedback questionnaire. 3. Video of 3 participants using LINA in a ‘Userlab’ facility using a Think-Aloud protocol. 4. Learning log of participant’s actions and reflections. 5. Researchers observation notes taken during videoing. 6. Transcripts of audio interviews following Userlab testing.

8. Method - sample * IET - Institute of Educational Technology

9. LINA Findings

10. Engage with question Type my answer Confidence-indicator tool How my mark is calculated Select my confidence level

11. Reflection, Justification, & Score ‘‘…it just introduces a bit of an edge to it in a gambling sort of sense’. [C-98] • ‘…it makes you think about what you’ve just said… It makes you evaluate... how well you think you’ve done’. [D-78] ‘…if I saw little gold stars or something…I think that would motivate me’ [S-188]. Important as a prompt to reflection and justification of learner choice (Principle 2). • Score-value important for encouragement and motivation (Principle 5).

12. Learners and self-assessment • Learning-oriented approach encourages reflection (Principle 5). • ‘… when you’re highly confident and then you don’t get it right, that inevitably makes you stop and think’. [C-104] ‘It wasn’t so much about the score, it was about working out where you’ve got it wrong’ [D-64]. Achievement-oriented (score-driven) strategy still leads to reflection (Principle 2)

13. Confidence Indicator tool – how useful?

14. Confidence indicator: summary • Pausing to self justify their confidence level encouraged students to reflect on the evidence for their knowledge • Not all student were good at self assessment and this could affect their score • The score was valued as a motivator by some, others would value it more if it could be personalised to reward (applause, stars) rather than penalise • Such control could encourage more engagement

15. Learning log See my score Read my reflections Note any learning points, items to revisit, reasons for next choice of Qs

16. Log: inserting own reflections • Prompted before leaving each Qs to input comments Qs difficulty Action points • The newer the learning, the more they might insert Reasons for choices Comments on own errors lack of clarity in Qs wording Feelings

17. Log: review learning pathway • ‘ I think if you know you’ve got that facility to monitor as you’re going along, that’s useful.’ [D] Monitor own notes (not score or systems data), retrace actions, support follow-up Personal choiceto use it more likely after incorrect answer (to ‘close the gap’) Needed familiarisation to adjust to proactive role fostered feelings of being in control

18. Log: review learning pathway • ‘If you’ve got something wrong, it makes you address it, and look at it again, and that’s how you learn’ [D] Other uses: print out, copy and paste details into forums • On next visit would like prompt of her previous reflections

19. Learning log – how useful?

20. Log: summary • Information in one place can aid self-regulation • Puts learner in control of providing high quality feedback themselves • Skills and self-awareness needed • Less experienced learners may not produce high quality reflections worth revisiting • (Untypically for CBA) responsibility for the quality of the feedback lies with the learner, rather than the teacher • Such control/responsibility may suit more experienced learners

21. Accessing question-based feedback • ‘external’ pre-authored feedback • to support ‘closing the gap’ Correct/incorrect What the correct answer is & why Explanation & tips Reference to relevant course materials

22. Feedback: why not referred to? Not needed for a confident learner: when clear what mistake is • ‘I thought ‘I’m wrong, what did I do wrong’, and then I go back myself and try and analyse it myself to find out what went wrong’ [C-3] • ‘Whatever the situation, if the end result is that I’m going to get a score, and that’s going to make a difference to e.g. my course result, that’s going to be a very important driver for me’ [C-52] • Achievement oriented If answer is correct, additional feedback has no value

23. Feedback: why/when referred to? Less confident learners • S: ‘apprehensive’ & ‘methodical’ • D: ‘systematic’ & ‘[tending] not to cut corners’ Learning-oriented • ‘when I got that question wrong I could then read and think that’s why, that was where you were coming from, and it was different to where I was coming from.’ [S-52] Formative: receptive to instructional role, transferability Tips should be easily distinguishable from main feedback • Approach dependent on readiness for new/ additional information

24. Feedback: why/when not referred to? Learning-oriented • ‘I would look on it as a learning thing, the fact that I’d actually worked through and got the question right at the end.’ [D-3] Wanted simply to know it was correct (not focusing on score) • ‘…as long as the way I’ve done it gets a correct answer, that’s all I’m interested in.’ [S-4]. Wanted consolidation, not confusing additional methods of doing the same task

25. Feedback: summary • Learning oriented and confident > ready to read and act > value feedback to help close gap/ extend learning • Familiarity with environment needed to encourage use and high quality self-generated feedback • Achievement-oriented (score driven) and confident > personal strategies so ignore feedback> not needed to close gap/additional learning is not sought

26. Feedback: implications • After correct answers fuller feedback not necessary? • Score-driven students may circumvent formative learning opportunities as they focus on summative goals (Miller 2008) • Learners need skills and awareness to self-generate high quality feedback • Learners value more personalised control over feedback to help make selective use based on its • Timing • Relevance • Quality

27. References • Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports student learning. Learning and Teaching in Higher Education, 1(1), 3-31. • Miller, Tess (2008). Formative computer-based assessment in higher education: the effectiveness of feedback in supporting student learning. Assessment & Evaluation in Higher Education, 34(2), 181-192. Retrieved August 22, 2008, from http://www.informaworld.com/10.1080/02602930801956075 • Nicol, D. and Macfarlane-Dick, D. (2006) 'Formative assessment and self-regulated learning: A model and seven principles of good feedback practice', Studies in Higher Education, 31 (2), pp. 199-216. • The REAP (Re-engineering Assessment Practices) project http://www.reap.ac.uk/ • Sadler, R., D. (1998). Formative assessment: revisiting the territory. Assessment in Education, 5(1), 77-84.

28. Forthcoming publication • Nix, I., & Wyllie, A. (In Press) Exploring design features to enhance computer-based assessment: learners’ views on using a confidence-indicator tool and computer-based feedback, BJET, doi:10.1111/j.1467-8535.2009.00992.x LINA demo and project website • http://www.open.ac.uk/colmsct/activities/details/detail.php?itemId=461cd0ac48c9d Contact Ali Wyllie - a.j.wyllie@open.ac.uk Ingrid Nix - i.nix@open.ac.uk