Integrated Programme Evaluation: ‘c apturing the student experience’

1. Integrated Programme Evaluation: ‘capturing the student experience’ JSWEC 2008 Workshop David Gaylard & Sam Baeza Acknowledgements: Phil Verrill (Head of Centre for Learning & Teaching)

2. Traditional evaluation methods • Tend to concentrate on the mechanics of learning & teaching • ‘Modulisation’ of current curriculum's can fragment the student experience, promote a ‘surface’ approach to learning (as opposed to a ‘deep’ approach) and may hamper integrated learning. • Students complain of ‘evaluation burn-out’, as a consequence, the quality and amount of evaluation data declines. • have been described by some as a ritual and not always regarded as a serious matter by either students or lecturers • students do not believe that universities take their feedback seriously, shown by poor response rates of evaluation forms, which in turn affects their reliability and validity • Provides little data that can actually be used for reflective development for teaching staff. • A growing recognition within HE that the student is the ‘consumer’ so their voice is increasingly important.

3. Student comments on existing evaluation process Just tick the boxes, it doesn't let you explain. I often feel we evaluate content & entertainment not the learning. Dull, no one cares! Doesn’t let you express yourself and they’re never totally anonymous. Current method is really irritating – we just write anything in order to finish the forms quickly. I sometimes feel that nothing will improve from our feedback. Having to do them for every module is a joke – it takes up so much time and feels as though I am repeating myself.

4. The whole programme evaluation = a challenge

5. Aims of Whole Programme Evaluation: • Creates a more transparent and holistic evaluation culture. • Takes as its starting point the centrality of the student voice. • Aims to enhance integrated and life-long student learning through reflective staff development. • Encourages student to give thoughtful and honest feedback that they value as much as the staff. • Provides a ‘rich picture’ of the student experience that enable teachers to reflect more deeply upon their own practice & consequent effect on student learning.

6. The whole programme process Involves a whole year group and isconducted face to face with students over a period of 2- 3hrs. 3 elements to the approach: 1. Student Questionnaire re: ● aspects of the modules that helped them learn; ● evaluation of the current evaluation methods; ● expectations & level of fulfilment of the course; ● students rank concerns in order of importance re: the first two areas BREAK: during which questionnaires are scanned for major themes under each heading. 2. Questionnaire discussion on issues raised in responses, enables clarification , probing of concerns to establish better validity and reliability of data, agreement from the whole group in terms of what was being reported. 3. Students asked to rate staff (on a 5point scale) in terms of what staff did to help them learn and what they did to prevent them from learning. Students asked to justifywhat they were saying in terms of student learning - avoiding an ‘academic beauty contest’.

7. How is data used • After data analysis a report is produced, detailing major issues for general learning and teaching development. • No data on individual lecturers revealed. • Individual feedback to lecturers on their student ratings given. (i.) as raw data for lecturers to reflect on; (ii.) a follow up session to discuss thoughts on data & any developmental plans made or support needed.

8. Student comments following a whole programme evaluation This process is better as we can be more detailed. This is much better because there is no pressure to rush I feel you are listening more to our feelings and queries. Evaluations could be improved by using forms such as this taking directed evaluation rather than just random comments I find it much more valuable and easier to voice my opinion in group meetings.

9. Some thoughts? • Are students competent enough to make such judgements? • Are ratings influenced by teachers’ popularity rather than their effectiveness?

10. Its applications There are a number of ways to access the student voice: • Mid – module evaluations or end of module evaluations • Cross – module or cohort evaluations • Personal interaction (‘coffee & cigarettes’ approach) • Practice placement evaluations involving students & practice assessors • Whole programme reviews

11. Programme Evaluation: Key Learning Points • This process requires good trained & empathetic facilitators • staff and students must have confidence in facilitator • Related to development not individual performance • Ownership of the process by students & staff • Prompt feedback to students and staff • Provides an effective mechanism for capturing data to make real programme development • The process can be seen as cost - beneficial, transparent, confidential & detached from any managerial agenda relating to performance • Efficient from a students’ perspective

12. The Challenges • Time consuming: requires a 2nd person to capture data during the session. • Depends upon the skill of the facilitator / evaluator • Requires the faculty, teacher, student to ‘buy into’ and engage with the process • Train student reps as evaluators to make process more time efficient • Teachers may withdraw - if uncomfortable. • Possible misinterpretation of themes / problems. • Danger of concentrating on ‘ranking’ instead of developing teachers attitudes towards feedback.

13. References • Campbell C. et al (2007) ‘Hearing the student voice: promoting & encouraging the effective use of the student voice to enhance professional development, teaching & assessment within higher education’ Final Report, Napier University, Edinburgh, Pub. • Crisp B.R. Lister P.G. (2002) ‘Assessment methods in social work education : a review of the literature’ in Social Work Education, Vol.21, No.2, 2002 p.259 -269 • Darby, J. A. (2007). "Evaluating course evaluations: the need to establish what is being measured." Assessment and evaluation in higher education 32(4): 441-455. • Morley, L. (2003). Quality and Power in Higher Education. Maidenhead, Open University Press. • Richardson, T. E. (2005). "Instruments for obtaining student feedback: a review of the literature." Assessment and evaluation in higher education 30(4): 387- 415. • Ticker T., Rangecroft M. & Long P. (2005) ‘Bridging the gap: an alternative tool for course evaluation’ in Open Learning,Vol.20,No.2 June p.185-192 • Verrill P. (2007) ‘Voices Odd? Using student voice for reflective staff development’, Hearing the student voice, project case study. • Useful website:www.napier.ac.uk/studentvoices