Understanding Students' Learning Gains: Insights from UK Distance Learners

1. Simon Cross, JekaterinaRogaten, DuyguBektik, Denise Whitelock, Bart Rienties Institute of Educational Technology, The Open University SRHE Conference / 7 December 2017 How do students understand and interpret learning gains? Perspectives from in-depth interviews of UK distance learners

2. Using data from interviews undertaken in May 2017 at the Open University UK, this presentation will probe the relationship between how students understand and interpret the learning gains they experience and the meaning and significance they give to metrics in use for measuring learning gain. PRESENTATION OVERVIEW ABC Learning Gains Project

3. Using data from interviews undertaken in May 2017 at the Open University UK, this presentation will probe the relationship between how students understand and interpret the learning gains they experience and the meaning and significance they give to metrics in use for measuring learning gain. PRESENTATION OVERVIEW ABC Learning Gains Project

4. Learning Gains As TEF develops we will incorporate new common metrics on engagement … and learning gain, once they are sufficiently robust. Government Green Paper Policymakers are now called on to provide evidence to inform questions such as: How do students’ knowledge, skills and work-readiness change and improve through their experience of higher education?... Traditional measures for evaluating student performance remain essential …but they do not provide all of the evidence needed to address these questions. And this evidence is more important than ever… HEFCE Average gain score needs to be interpreted with caution and there remain many outstanding issues that need investigating. Pascarella et al. 2011.

5. 1 How do students understand and interpret learning gains they experience? 2 How do students understand and interpret measures of learning gain currently in use? 3 To what extent can these two be reconciled? Research questions How do students understand and interpret learning gains?

6. Semi-structured interviews with nineteen part-time distance learners resident in the UK lasting 30-60 minutes Student perception of gain and progress Cognitive, behavioural and affective change Relationship between grades and progress Study expectations Work relevancy Methodology and participants How do students understand and interpret learning gains?

7. First stage Interim analysis based on review of interviewer notes, second listen to interview and coding of interview transcripts. Methodology and participants How do students understand and interpret learning gains? Attainment grouping as measured by assessment marks Progress grouping as measured by assessment marks Faculty Gender Student 8 Female High Lowest STEM Student 19 Female Low Medium Arts/Soc.Sci Student 5 Female Low Highest Edn/Lang Student 10 Male High Medium Arts/Soc.Sci Student 12 Female Low Lowest STEM Student 18 Female High Highest STEM Sources of quotations used in presentation

8. LEVEL OF STUDY LEVEL OF STUDY RELATIVE VALUE OF GRADES Emerging themes How do students understand and interpret learning gains? WORK-LIFE OWNERSHIP OF LEARNING

9. Level of Study ‘Student 8’ Looking back Looking ahead Current module How do students understand and interpret learning gains? [Initially] it was very much like study for study purposes [but] about a year and half into it, my mind-set changed, it was like I enjoy what I’m doing and it’s giving me something tangible. This year was the first year when … it reflected back on my day job. The outcome was a decision to increase study intensity… I [feel] that I have the knowledge, it’s the time that I was missing… I don’t think for me it’s grade itself is… it’s actually the knowledge. [Getting the grade] is what’s needed this year for me not to give up so that I had an opportunity to do the exam and face the next step. I have decided that I will care about my grades next year and try to get the highest scores I physically can just to see [how well I can do] if I don’t have that additional 60 credits.

10. Relative Value of Grade Explaining lower grades • Reassessment and reprioritisation • Focus on elements relevant to own interests • Expended the oxygen of prior learning • Expectation that modules will become tougher and grades harder to achieve • Module does not directly build on last • Comparatively less effective learning design • Greater study workload How do students understand and interpret learning gains? There were things in the first two modules that I already knew [but] when I come to this year [then] I would say probably 95% of what I’ve learned … I’ve never heard of. So, I think this has been a real turning point for me. [This year] was my turning point… Your TMA (continuous tutor marked assessment) is not everything. It’s not, it’s supposed to be what you actually physically know yourself inside… I think that’s really important.

11. Owning the process Initiating discussion for feed-forward Counter-intuitive deeper engagement How do students understand and interpret learning gains? I sort of challenged [the TMA score], not ‘challenged’ it really, but asked more questions. [I’ve learned] don’t be afraid to.. I thought I possibly would have done better, you know maybe 5 or 10 marks here and there so I have challenged. [The tutor response] helped me… I like to read it, dissect it [and] this year the feedback I’ve got has included things that I possibly might have missed. Despite the drop in marks, I still think my understanding has deepened and now … I feel more engaged in [my learning] actually, not just ticking off what I need to do

12. Work-Life Reference (Points) Becoming aware Becoming confident How do students understand and interpret learning gains? I notice that my approach to things outside of the academic context is different … I judge [gain] on the assignment side and the out-of-academic context side …as well. The way I think, the way that I possibly act at times, my life feels different now … I can talk confidently to people. It’s just when you learn something you [then] become aware of, for instance, either news, work itself, everyday life; which again kind of changes your perspective and then it allows you to properly build your own confidence because you understand things

13. Reconciling perceptions of learning gain? Many factors impact on the quality of assessment that students choose to submit. Students’ perception of ‘learning gain’ change as they progress. Level 2 seems to be a key turning point. Students and assessors can interpret and give meaning to the same grade in different ways Whilst assessors set the assessment criteria, it is the student who decides its relevancy to their learning. Becoming an independent learner means taking ownership of the meaning and use-for-learning of the assessment process Students (co-)determine the significance of a grade through a process of contextualisation across study, social and work lives. How do students understand and interpret learning gains?

14. Concluding remarks Significance and importance of qualitative research Short and medium term goals An A+B+C Approach How do students understand and interpret learning gains? Email: simon.j.cross@open.ac.uk