The effect of generic and episodic reflection on academic skills: an empirical study

1. The effect of generic and episodic reflection on academic skills: an empirical study Astrid Schepman*, Paul Rodway*, David Reynolds**, Gemma Coleman*, Peter Hartley*** *University of Chester **now Sheffield Hallam University ***University of Bradford email: a.schepman@chester.ac.uk

2. Acknowledgements • Funding: Higher Education Academy Psychology Network Mini-Project Grant (2009-10) • Assistance: Jordana Lambert, Julie Lloyd (University of Chester)

3. Reflection • Used widely, in psychology and beyond • Does it enhance performance? • Is it beneficial in other ways? • What evidence? • For practitioners AND students • Processes: introspection, analysis, self-evaluation • cf. Cognitive psychology • First try: high level cognition (executive function, self-regulation) – next slide

4. Our past research: brief reflectionUniversity of Bradford, with Peter Hartley and Rebecca Durrans • Brief reflection immediately after an event • Planning / executive function task • 10 min task – 10 min reflection (or control) – 10 min task • Improvement after reflection without prompts • But: reflection prompts: Questions: scaffolding / task analysis: Did not improve performance • Fragmentation of thought? Distraction from most important issues? Blocking effective reflection?

5. Current research • Reflection on key academic skills: • Note taking, exam essay writing • More implicit cognitively, more complex, more real • Can brief reflection show tangible benefits in these tasks? • (as in our previous research, which tapped into executive function?)

6. “Generic” vs. “Episodic” reflection • Which is better? • Reflect on general experience (“generic” reflection) • Reflect on specific recent event (“episodic” reflection)

7. To scaffold or not to scaffold? • Students can find it hard to reflect: so prompt? • Current research: provide list of ideas • Question approach not effective in our previous research

8. Method: Participants • 120 undergraduates University of Chester– all years • 67% psychology single or joint honours • 33% others, primarily social sciences / arts • Average age: 21 • Average self-reported grade: 63.6

9. Method: Materials and Procedure • Clips: Carbon Capture, Dark Matter • no prior knowledge • Generic reflection: • Reflect (or control) – Clip & notes – Answer • Films 50-50 • Episodic reflection: • Clip & notes – Answer – Reflect (or control) – Clip & notes – Answer again • Order of films counterbalanced

10. Reflection conditions: Write about holiday trip and / or news item and / or sporting event. • 1) control (write, reflection blocked) • 2) unprompted (general request to reflect on note-taking and exam essay writing) • 3) with “prompts” (ideas in list – see next page) reflect on (think about) two key skills... ...how you normally do these (G) / have just done these (E), ...how to improve ... (G) (E: added: ...when you do the task again). ALL: Write down thoughts for ten minutes

11. “Prompts”: Choose from ideas or use own: Note taking: Simultaneous listening and writing, not missing information, maintaining attention / motivation, helping you remember, helping organise information, recording important points, highlighting most useful information, using abbreviations or codes, using special layouts to help your learning, keeping “to do” notes Short answer writing: Checking you have understood the question, planning the answer, structuring the answer, citing sources, remembering information, screening information for relevance, applying all your knowledge to the answer, planning your time effectively, writing clearly, checking you have answered the question, having a summary or conclusion (participants given extra time to read through these)

12. Primary dependent variable • Grades (double-blind marked) • University of Chester marking scheme • (0 -100)

13. Results: effect of reflection on grades: Generic reflection Analysis of Variance: not significant

14. Results: effect of reflection on grades: Episodic reflection Analysis of Variance Main effects and interaction not significant

15. Discussion: Grades • No significant difference / improvement in grades • Fine-grained marking showed no condition differences either, but correlated well with overall grade (consistency / validity) • No evidence of tradeoffs: only positive correlations between elements

16. Post-test questionnaire“Generic” reflection • Taken: Subjective ratings of performance, behaviour, attributions • Results: Minimal detectable effects of conditions on ratings.

17. Post-test questionnaire “Episodic” reflection • Same measures taken: session 1 AND session 2: differences: • Students gave themselves worse ratings in second answer for performance if they had reflected. • But student gave themselves better ratings in second answer on “planning, timekeeping” if they have reflected. • Control condition stays same from first to second answer.

18. To scaffold or not to scaffold? Subjective rating results prompts: • Prompts rated as very helpful in production of reflective text and in task performance • But no evidence in actual performance data, nor in self-rating when asked indirectly • ! Care needed when evaluating impact of reflection interventions

19. The reflections:Categorisation coding • Evaluating performance: • Generic reflection, prompted reflection: more complex content (i.e. more content categories used) • + additive interaction • “narrowing” of content for episodic • Ideas for improvement: • Episodic: more ideas on how to improve, but no effect of prompting, so ideas come from recent experience

20. Discussion • Participants who reflected thought their performance got worse (Episodic reflection) • In fact their performance was unchanged • Metacognition: Not useful “recalibration”: Already underconfident on prediction of own performance • Predict around 50, obtain around 58 • Participants who reflected thought their planning got better • In fact no evidence in structure or number of key ideas • Reflections: recent experience and prompts have separate benefits

21. So: is reflecting on academic skills useful? • No evidence of benefit of brief reflection on note-taking / essay writing skills as measured by performance: Why not? • Variation due to other factors? • Impact not immediate like effect of brief reflection on planning / strategy (Bradford)? • Too complex, too implicit, can diagnose but not change in time available? • More research needed

22. Take-home messages • Careful when we ask students to invest time in reflection: It may not always benefit them • Reflection needs much more research: widely used, poorly understood • Psychology has much to offer in helping us understand reflection more precisely