Experiences of audio assessment feedback: Staff and Student experiences

1. Experiences of audio assessment feedback: Staff and Student experiences Carol Ekinsmyth Department of Geography University of Portsmouth (carol.ekinsmyth@port.ac.uk)

2. Why audio feedback might be important • Teaching and learning in higher education is often confounded by dominant cultures and quality audit frameworks that prevent the achievement of these desirable goals: • The sharing between ‘teachers’ and ‘learners’ of integrated knowledge structures (Kinchin, Lygo-Baker and Hay (2008) • The breaking down of “pedagogic resonance” (Trigwell and Shale (2004), Kinchin et al (2008) • Helping students arrive at and move beyond “liminal space” and develop understanding of “threshold concepts” (Meyer and Land, 2005)

3. Assessment and feedback • Researchers agree that the majority of student effort (learning?) is geared towards assessment (Carless et al, 2006; Brown, 2005; Gibbs, 2006) • Carless et al (2006) encourage us to engage in “learning orientated assessment” • Smith (2007, 11) argues: “If assessment is for learning, then feedback needs to focus on what the student could do differently and how to apply that learning to future work.” • Price (2007) and Baumard (1999) argue that the trend towards over-reliance on written feedback in HE crushes the ability to effectively encourage tacit knowledge formation and concentrates instead only on explicit knowledge

4. Assessment and feedback • “Good feedback comprises not just commentary about what has been done, but suggestions for what can be done next” (Brown, 2007) And she argues; • “I believe that concentrating on giving students detailed and developmental formative feedback is the single most useful thing we can do for our students” (Brown, 2007) • Race (1999) argues “...feedback quality and quantity are probably the most important factors in enhancing student learning” • And yet, my experience suggests that Staff are reluctant to spend a lot of time on feedback ......

5. Why the reluctance over feedback? • Time pressures • Uncertainty about what feedback is for: • Kinchin et al (2008) talk about more fundamental barriers – complicity between teachers and students in “non-learning outcomes” “Correcting errors, providing guidance for future work, justifying the mark, satisfying quality assurance processes, diagnosing difficulties, the list goes on.” – Price (2007)

6. How can audio feedback help? • It can’t - if teachers are unwilling to use it • We perhaps need to concentrate on overcoming the barriers to teacher-engagement • Audio feedback seems especially suited to feed-forward as a supplement to assessment proformas that often concentrate on feedback • Used in carefully identified places in the curriculum, it may be a powerful tool in breaking down pedagogic resonance

7. Why? • Because it enables the students to get closer to the thought processes (and thus community of practice) of the academic than written feedback does • Face-to-face contact might not be better as it can be intimidating and the details often cannot be recalled by students (my research findings)

8. The Project • GEES/LTSN (JISC ‘Sounds Good’ project) • Aims: To evaluate the utility, efficiency, popularity and effectiveness (amongst students and staff) of audio feedback on assignments (range of types) for undergraduate students within the discipline of Geography.

9. Other reasons to bother • UK National Student Survey (2005-2008) shows that the area that students are least satisfied with is ‘assessment and feedback’ (Race and Pickford 2007) • The centrality of assessment to the student experience is widely acknowledged (Ribchester et al (2008), Brown 2005) • Research has shown that on average, 90% of all student study activity is directly related to assessment (Ribchester et al (2008)) • Feedback that academics provide may not be effectively used by all students (Higgins et al (2002)) • Academics and students have been shown to differ on their perception of what makes good feedback (Carless (2006)) • “…the literature on student experiences of feedback tells a sorry tale” (Handley et al (2007, p1) quoted by Ribchester et al (2008)) • Shriver (1992) has shown that hearing feedback helps writers appreciate the effects of their writing on the reader (Merry and Orsmond (2007)) • The technologies exist so why not experiment!

10. An example • My feedback – A , B Probably easier to leave feedback when the work is good rather than poor.

11. Reactions: Student experiences • Student audio files - Response D • Response A Response B Response D

12. Student Reactions: Positive thoughts • Key aspects of audio feedback that students liked were; • Its more detailed nature • Tone of voice (enriches experience) • Clarity – “You can take it in easier” • Not reliant on handwriting • Its more personal nature – “I listened carefully because it is more personal” • Its potential for feed-forward

13. Students taking more notice? “I think it is more articulate than written text. I think it allows you as a marker to articulate your points a lot better than a written line of text does - and it allows you as a student to take the feedback on board more. I mean, I normally dismiss the written feedback form if I’ve done really badly” “And you can’t read ahead. You have to listen to it all. You don’t know what’s coming (general laughs). When you’ve got a sheet you can skim-read it ….”

14. Student Reactions: Positive thoughts “I find it interesting to hear how you are criticising my work (tone)” “I really liked it” It is definitely more memorable because it is a different way” “It was the most feedback I have ever had on a single piece of work” “Written feedback is often too brief and difficult to read” “Overall I think it is extremely useful” “I take information in better when it is in a spoken form” “Very easy to use”

15. Student Reactions: Negative thoughts • It doesn’t replace the written feedback • Nothing visual to look at • Can be too personal for comfort • You can’t answer back/ engage in dialogue

16. Student Reactions: Negative thoughts “It was a bit more personal, therefore I feel harder to take the criticism” “Personally I do prefer hard copy – I’m a bit old fashioned like that … so I do think you shouldn’t loose hard copy of feedback – you can keep that for life then” Strange at first – “felt like my lecturer was in the room with me” “It would wear off. It can’t be used all the time – or it won’t go in it won’t be registered at all well” “My main dislike was that I had nothing visual to look at“ “..perhaps made you more nervous.. Builds a bit of tension” “I found that some of the feedback was very critical and it upset me more as an audio feedback than a written feedback”

17. Staff Reactions: Focus group discussion • There was a reluctance to like this method – why? • Reactions focused on issues of: • Exposure and feedback quality • Feasibility/practicality • Appropriateness • Best applications • Context limitations and litigation • Language • Skill acquisition

18. Staff Reactions: Exposure and feedback quality • Do we ‘hide’ behind written feedback conventions?

19. Staff Reactions: Exposure and feedback quality • Do we ‘hide’ behind written feedback conventions? “You can hide more in written feedback – you are much more explicit when speaking something so you need to think much more carefully” “When I was doing audio, I couldn’t hide anything… it made me think more about what good feedback constitutes” This isn’t to suggest that teachers are intellectually lazy, but under inevitable time pressures during summative marking processes, it was felt that feedback short-hand comments require less time and effort than comprehensive spoken feedback. Do feedback proformas ease or exacerbate this?

20. Staff Reactions: Exposure and feedback quality • Written feedback, especially that coded onto generic feedback sheets perhaps leads us to write feedback in a form of short-hand that needs to be deciphered by the students. • Do we teach students to decipher this short-hand well enough? • Staff can ‘hide behind’ this short-hand which, for reasons of lack of time or the repetition of the process, reduces the need for great qualification of opinions or development of comments. • In audio feedback, you are able to develop this feedback with “for example.............” “You need to contextualise these findings by referring to debates in the academic literature” “You do not develop your arguments thoroughly enough” “Your discussion is too anecdotal”

21. Staff reactions: A consensus • Feed-forward rather than feed-back is where the greatest strength of this method lies • Not everyone however is convinced of the need for thorough feed-forward – why? • Unitised curricula structures • Goal-oriented teaching and learning structures • “Economies of practice” for teachers and students (Kinchin and Hay 2007, 98) • Teachers and students complicit in a cycle of “non-learning” (Kinchin and Hay 2007, 98)

22. Staff reactions: Feasibility and practicality • It takes longer if you are going to do it properly (i.e. Use the method to its greatest potential) ““My gut feeling will still be … phew…. it’s a lot of work” “I don’t think it will ever be widely applicable” “At the moment based on my own experiences, I think the same, its beyond us but it may be because it is new, its learning a new language and new skills”” “I think that if had undertaken better preparation for my 5 minutes of audio feedback, by preparing the comments in advance, then the work necessary (and time taken) would have been much greater than what I do when I’m giving written feedback, and we do need to keep coming back to that“

23. Staff reactions: Practicality “You do need a quiet room to sit down … we work in a fragmented way and slot in marking … to make it efficient. Sitting down to record a narrative was problematic. To have a continuous narrative for 5 minutes was difficult.”

24. Staff reactions: Practicality and disembodiment “The practicality of it, to me, is my main barrier – I think ideologically and pedagogically there are some benefits but the shear practicality, the fact that we almost disembody the feedback from the written text and then to reconnect – I’m not sure there is anything lost but I expect there is… but simply the recording of it, thinking of what you are going to record, it is a two-stage process whereas when writing, it is all in one”

25. Staff reactions: Language and Litigation “It is much more difficult if you are being natural to control the language that you are using. Now in our feedback we have to work to language criteria – it’s a very controlled environment – excellent is a first, very good is a 2.1 – so audio feedback, it’s almost impossible to work within those limitations - and if you do work to them, it is pointless.” “The audio feedback has to be different from the written in that it can display some emotion or passion” Do we need to accompany audio feedback with written guidance as a disclaimer?

26. Staff reactions: Applications “We have sanitised our written feedback so it is probably less memorable. There’s clearly a role for it, you can motivate and stimulate much more easily it is much more memorable.” “I’d be happier giving feedback in a formative sense released from the constraints of grade point criteria and the associated language” “I’d be a little wary of replacing written comments with audio feedback – not least from a quality perspective, records for External Examiners and so on” Formative feedback Feed-forward But what of the novelty value?

27. Staff reactions: Re-accessibility “The re-accessibility aspect is important. They can go back to parts of the written feedback but would they listen to a 5 minute file again if they only needed a bit? Also, they have to listen to the negative criticism again if there is some, and they might not want to do this and thus may not be able to bear to listen again” Will students keep it and listen to it again? How do they keep it with the work? Do they need instructions?

28. Conclusions/ Recommendations I • Excellent for formative feedback/ feed-forward – perhaps not so useful (or necessary) for summative feedback. • Staff were more cautious than students, remaining unconvinced overall of the cost-benefits but recognising great strengths and potentials if used judiciously. • For summative feedback, neither staff nor students recognise audio feedback as a replacement for the rule-governed written feedback. • Students in particular felt that audio feedback provided a much more detailed and richer account of the strengths and weaknesses of the work.

29. Conclusions/ Recommendations II • Students felt that hearing feedback was more effective and memorable than reading it but both staff and students felt that the sporadic use of audio feedback was important in this respect. • Thus the judicious use of use of the audio medium at the most appropriate moments would be the best approach. • Staff had concerns about the need to be very careful about language and tone in audio feedback, and felt that a separation of intent between written feedback and audio feedback would be useful.

30. In a nutshell • Excellent for feed-forward • Saves time in delivering detailed feed-forward • Not time-efficient for vast piles of scripts • Best times to use method in the curriculum/degree programme need identifying • A key problem is breaking through established economies of practice of teachers with regard to feedback • Central to this is a re-evaluation amongst practitioners of the goals, possibilities and importance of feedback

31. Thank you Carol Ekinsmyth Principal Lecturer Department of Geography University of Portsmouth

32. References • Baumard, P (1999) Tacit Knowledge in Organizations, London, Sage • Brown, S (2007) Feedback and feed-forward, Centre for Biosciences Bulletin, 22, Autumn 2007 • Brown, S. (2005) Assessment for learning. Learning and Teaching in Higher Education 1, 81-89. • Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31, 219-233. • Carless D, Joughin G and Mok M (2006) Learning-orientated assessment: principles and practice. Assessment and Evaluation in Higher Education 31, 4, 396 • Gibbs G (2006) How assessment frames student learning. In C Bryan and K Clegg (Eds.), Innovative Assessment in Higher Education, Routledge, London. • Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more: Student’s experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc • Higgins, R., Hartley, P., Skelton, A. (2002).The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27(1), 53-64 • Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more: Student’s experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc

33. References • Kinchin I, lygo-Baker S and Hay D (2008) Universities as centres of non-learning, Studies in Higher Education 33, 1, 89-103 • Merry, S., Orsmond, P. (2007) Feedback via MP3 audio files. Bioscience HEA Academy Centre for Excellence Bulletin, Autumn 2007 • Meyer J and Land R (2005) Threshold concepts and troublesome knowledge (2) – epistemological considerations and a conceptual framework for teaching and learning, Higher Education, May • Price M (2007) Should we be giving less written feedback?, Centre for Biosciences Bulletin, 22, Autumn, 9 • Race, P., Pickford, L. (2007). Making Teaching Work: Teaching Smarter in Post-compulsory Education. Sage, London. • Race, P (1999) Enhancing Student Learning, Birmingham: SEDA • Ribchester, C., France, D., Wakefield, K. (2008) ‘It was just like a personal tutorial’: Using podcasts to provide assessment feedback, paper presented at the Higher Education Academy Conference July 2008 • Shriver, K. (1992) Teaching writers to anticipate reader’s needs. Written Communication 9(2) 179-208 • Smith B (2007) Is assessment really for learning?, Centre for Biosciences Bulletin, 22, Autumn, 11 • Trigwell K and Shale S (2004) Student learning and the scholarship of University teaching, , Studies in Higher Education 29, 4, 523-36

34. Appendix Practicalities

35. Phases and progress • Phase one – Experimentation • Using audio feedback in a range of ways (individual feedback, group feedback, generic feedback, ‘bite-sized’ feedback • Trialling WIMBA as the enabling platform

36. Phases and progress • Phase 2 • Staff experiences of using audio feedback recorded in a videoed focus group. Video to be podcasted for use in staff-training workshops. • Student experiences and evaluations of the method elicited through Wimba voice email, focus groups and questionnaires.

37. Phases and progress: Future plans Phase 3 Investigation into where in the curriculum audio feedback can be most effective Phase 4 Evaluation and dissemination (including production of staff-training materials)

38. Constraints • With limited time and large student numbers, any method will need to be measured in terms of a cost-benefit analysis • Giving more detailed feedback takes time – so what methods are available that might help in this respect?

39. Phil Race (2008) – on feedback(www.phil-race.co.uk retrieved 12.12.08) Two of his key recommendations are: • Making sure they get feedback quickly enough so that they still care about it • Ensuring that they get plenty of feed-forward, so they can make their next piece of work better

40. Audio feedback – my method • Read work, make notes against the assessment criteria. • Also make notes on each section of the work (perhaps paragraph by paragraph for a traditional essay). This might be in the margins of the work. • Take a note of overall feedback. • Make a voice recording of this feedback. • Send to student. • Provide summary notes on a conventional marking sheet for return to the student.

41. Staff Reactions: My experiences • Can be daunting at first leaving the recording but you quickly gain confidence and do not feel the need to listen to the message • Despite worries about tone of voice or in authenticity of sound, if you are natural and speak as you normally would, the recordings are well-received • You get quicker at the whole process – at first you think it will never work. Having said this, it is more time-consuming than methods I am used to (but the quality of feedback is much greater) • In spoken form, I could go into depth about issues and make suggestions that I simply wouldn’t have time (or inclination) to do in written-form. • Minute for minute therefore, it is much quicker to speak than to write. • You need to record the feedback after you have read an

42. Audio feedback- Colleague A’s method • Read work and record feedback instead of writing it on marking sheet – trying to take no longer than he would normally take in conventional marking/ feedback production

43. Audio feedback – Colleague B’s method • Provide feedback in a PowerPoint presentation – scanning in sections of the student work and using ‘narration record’ to provide explanation of points.

44. Hardware & Software • The recordings could be made using any MP3 or digital recording devise but sending files then takes time. • VLE/Wimba Voicemail can be used and for this, only a headset is needed. • PowerPoint can be used for audio-visual feedback

45. Wimba audio archive