Transforming assessment through the tenets

1. Transforming assessment through the tenets Sally Brown and Brenda Smith 23 April 2013

2. This session aims to offer: • Some thoughts on the nature of assessment as a transformative process to enhance student learning; • An overview of the six tenets; • A review of how tenets one and two integrate holistically with the other tenets; • Some practical suggestions to support your work based on tenets 1 and 2; • Opportunities to develop your work and share ideas and issues so you can start to make some meaningful changes at your institutions.

3. Background to ‘A marked improvement’ • Stems from firm evidence that assessment is not‘fit for purpose’ (Race 2010, Brown, 2010); • Aims to take a radical approach, recognising that it is time for significant reappraisal of assessment policy and practice (ASKe, Weston Manor Manifesto); • Builds on expertise, perspectives, and previous work; • Takes an evidence-informed approach; • Encourages for assessment to be seen as an integral part of the learning experience.

4. Why change is needed • QAA subject reviews • Burgess Report - “system no longer fit for purpose” (p5) • QAA - “…it cannot be assumed students graduating …. will have achieved similar standards” (2007) • Media accusations of dumbing down and grade inflation • National Student Satisfaction Survey • “There is considerable scope for professional development in the area of assessment” (Yorke et al, 2000, p7) • Students become more interested in the mark and less interested in the subject over the course of their studies (Newstead 2002, p2).

5. Tenet 1 “The debate on standards needs to focus on how high standards of learning can be achieved through assessment. This requires a greater emphasis on assessment for learning rather than assessment oflearning”

6. Why change is needed • Our current systems focused on marks and grades are not working: • Belief that it is possible to distinguish the quality of work to a precision of one percentage point (Elander & Hardman, 2002); • Belief that double-marking will ensure fairness and reliability (Polani, 1998); • Belief that consistency can be achieved through conformity, and simple numerical rules (e.g. level 1 essay 3,000 words, level 3 essay 5,000; or no more than two pieces of assessment per module); • The combination of scores, which obscures the different types of learning outcome represented by the separate scores; • The distortion of resulting degree classifications by the application of idiosyncratic institutional rules (e.g. Armstrong et al, 1998).

7. Assessment for learning Adapted from Sambell, McDowell and Montgomery (2012, p5)

8. We need to focus on high standards of achievement and to clarify what we seek • “Even when lecturers say that they want their students to be creative and thoughtful, students often recognise that what is really necessary, or at least what is sufficient, is to memorise” (Gibbs, 1992, p10) • We need to emphasise for students the importance of spending rich and purposeful time on task, not just going through the motions

9. Key questions: To what extent and how do you evidence this? • Are you confident that assessment tasks demand high standards of learning? • Is assessment for learning given emphasis in relation to assessment of learning? • Do you ensure an appropriate balance between formative and summative assessment? • Is assessment and feedback planned within and across a programme to ensure appropriate student preparation and practice before summative assessment takes place? (From ‘A marked improvement’)

10. Tenet 2 “When it comes to the assessment of learning, we need to move beyond systems focused on marks and grades towards the valid assessment of the achievement of intended programmeoutcomes”

11. Why change is needed • Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004) • “We can know more than we can tell” (Polanyi, reprinted 1998, p.136) • Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987) • There is a cost (in terms of time and resources) to codifying knowledge which increases the more diverse an audience’s experience and language (Snowdon, 2002) • Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991).

12. Key questions: To what extent and how do you evidence this? • Is their an emphasis on assessment for learning over systems focused on marks, grades and reliability? • Does the assessment design process ensure valid assessment of the intended learning outcomes? • Is there a trade off between reliability and validity of assessment? • Are assessment decisions in relation to design, development and variety made within a programme context and focused on learning outcomes? • (From ‘A marked improvement’)

13. Tenet 3 “ Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently, diminish the learning experience and threaten its validity. There are important benefits of higher education which are not amenable either to the precise specification of standards or to objective assessment” (for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

14. Why change is needed • The most significant factor in student academic success is student involvement fostered by student/staff interactions and student/student interactions (Astin, 1997) • The only common factor in a study of departments deemed excellent in both research and learning and teaching is high levels of student involvement (Gibbs, 2007) • “Participation, as a way of learning, enables the student to both absorb, and be absorbed in the culture of practice” (Elwood & Klenowski, 2002, p. 246)

15. Tenet 4 “Assessment standards are socially constructed so there must be a greater emphasis on assessment and feedback processes that actively engage both staff and students in dialogue about standards. It is when learners share an understanding of academic and professional standards in an atmosphere of mutual trust that learning works best” (for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

16. Why change is needed • Important aspects of complex, high-level learning outcomes can only be achieved when students are allowed time to ‘come to know’ the standards in use by the community: • Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke 2004) • The achievement of high-level learning requires integrated and coherent progression based on programme outcomes • Where there is a greater sense of the holistic programme students are likely to achieve higher standards than on more fragmented programmes (Havnes, p. 2007) • Students need to engage as interactive partners in a learning community, relinquishing the passive role of ‘the instructed’ within processes controlled by academic experts (Gibbs et al, 2004)

17. Tenet 5 “Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of standards and monitor and supervise their own learning” (for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

18. Why change is needed • Changes in higher education (e.g. massification, reduced unit of resource, expectations of increased productivity in staff) threaten the ‘health’ of disciplinary communities and their ability to share and exemplify professional judgement. • There has been slow progress in the professionalisation of university teachers. • There has been limited attention paid to professional assessment practice. • Reliance on the external examiner system to mediate standards within the system is misplaced (Newstead and Dennis,1994). If some aspects of high-level learning can only be assessed using professional judgement then we need to ensure that judgement is indeed – professional!!

19. Tenet 6 “Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities” (for further discussion at later targeted forums, but here as part of a holistic discussion of the tenets as a whole)

20. Why change is needed • The current approach to assessment standards is inadequate; • The quest for reliability is getting in the way of learning; • We should be enabling students to achieve high level, complex learning; • Active involvement by staff and students in the learning community is essential to reach common understandings of assessment standards; • To achieve change there may need to be a review and evaluation of the allocation of time and resources within HE.

21. Task: working in institutional teams • Using the framework on the next slide, focus on tenet one to interrogate your own practices to date and where you can make improvements; • Identify between three and five ways in which you would like to make changes in assessment practice to make your assessment focus more on assessment for learning. • Prepare some summary bullet points on a flip chart sheet for sharing with other HEI teams.

22. Tenet 1 – Key PointsAssessment for learning

23. Task: working in institutional teams • Using the framework in the next slide, discuss with colleagues what you consider your top priorities to work on from the questions and suggestions provided there; • Identify ways that you would measure and evaluate changes as a result of the actions you plan to take; • Ask a spokesperson to be prepared to report back on this in plenary.

24. Tenet 2- Key PointsEnsuring assessment is fit for purpose

25. Programme learning outcomes should reflect what students should achieve • Making it clear to students what is expected of them; • Making it clear to teachers what students are expected to learn in their own and other modules;  • Helping teachers to select the most appropriate teaching strategy for the   intended learning outcomes e.g. lecture, seminar, tutorial, group  work, discussion, student  presentation, laboratory work. • Helping teachers to select the most appropriate assessment style  to assess the achievement of the learning outcomes, e.g.  • project, essay, performance assessment, multiple‐choice questions, exam; • Having a focus on programme learning outcomes – staff therefore need time to collaborate; • Are you confident that students being marked by different people or the same people at different times (inter and intra-tutor reliability) will achieve equivalent marks?

26. Methods of AssessmentAssignments and the learning opportunity they offer Communicate information and ideas in other than the written word Become more adept at writing in different forms, formats and ‘genres’ To understand the benefits and challenges of working collaboratively

27. Methods of AssessmentAssignments and the learning opportunity they offer Communicate information and ideas in other than the written word Seminar, poster, video, video conferencing or other multimedia Become more adept at writing in different forms, formats and ‘genres’ Design or proposals; book reviews; case report/study; web pages; journal articles To understand the benefits and challenges of working collaboratively Group problem solving; joint book reviews; team presentations; role plays; group exhibition

28. Balancing and timing formative and summative assessments University of Poppleton 2013 • Is the schedule of assignments and the feedback on these assignments made clear to students across the year? • Do you provide multiple opportunities for small low stake assessments with feedback? • Are your students given opportunities to prepare and practice? • Is there an appropriate balance between summative and formative assessment? • Is assessment and feedback planned within and across programmes and are staff given time to ensure this happens?

29. Figure 1: A student’s estimate of his weekly study effort (in hours) on a programme with no assignments and an exam in week 12 Gibbs (2010) Total study effort = 68 hours

30. Figure 2: A student’s estimate of her weekly study effort (in hours) on a program with three assignments due in weeks 4, 7 and 10 and an exam in week 12 (Gibbs 2010) Total study effort = 78 hours

31. Assessment for LearningDo your students always know what the criteria are for all work they are given?How do you help students to understand the criteria?Could the students re-write the criteria in their own words?

32. Key issues in feedback – develop the capacity to use feedback effectively • Students can’t read our writing • There is too much emphasis on grades and marks at the expense of learning • The feedback given is often not very useful and comes too late • Students are not actively encouraged to self reflect • Little or no use is made of peer/self assessment and feedback • Little dialogue takes place around feedback • Students have little opportunity to collate feedback over time and act upon it • Most feedback does not feed forward, it only tell students what they have done that is incorrect • Very little use is made of feedback as a normal part of the learning and teaching process • Staff vary in their approach to feedback

33. Some thoughts on promoting the engagement of students by helping them to get confident with assessing • Give students in groups a selection of essay or other relevant piece of work Excellent, Good, Pass, Near pass, Fail • Let the students decide individually which is which and then compare notes • In groups get them to tease out the differences between them and express them as criteria • Produce suggestions for writing an effective essay or relevant piece of work • Get them to write appropriate feedback comments • Exchange the comments with a different group and ask them to comment on the feedback. Is it helpful, would they know what to do differently, does it feed forward etc? • What suggestions would they give to the group to enhance the feedback comments?

34. How you might help colleagues develop the capacity to use feedback effectively • In small groups exchange your assignments and compare the comments received • Do you understand the comments, if not discuss what you think they might mean? • How would you respond to them? • What would you do differently next time if you redid the same assignmentand applied the learning to a new and different assignment? • Return marked work back to the individual but with no grade or feedback • Get them to mark their own work • Collect in their grades (Pass a sheet round) • 9 out of 10 students will be within 5% • Arrange to talk to those where the difference is more than 5% or one grade

35. Saving time – a range of approaches to feedback need to be in place • Giving generic feedback within 24 hours on the VLE • Offering face- to-face feedback with whole group/small groups • Using peer feedback in groups • Using technologies including: A sound file – Audacity Dragon software Statement Banks “Sounds good project” • Encouraging students to use self feedback • Helping students to recognise that feedback can come from many different sources (including themselves) and given in different ways

36. Involve the students- Feedback as a dialogue “When I get to sit down with my tutor and discuss my work – that’s probably the best form of feedback I can get”

37. Promoting the engagement of students across modules and programmes Learning Most significant feedback comments What did you understand these to mean? Analysing Things that I did that attracted positive comments Things that I did that attracted critical comments Any recurring trends or comments? Planning and feeding forward Things I can do to build on the positive feedback Things I can do to address the critical feedback (after Race, 2007).

38. Bringing this thinking together • All the tenets together provide a coherent agenda to bring about a marked improvement in assessment in your HEI; • The focus of the rest of the morning is to present your initial plans of action for enhancement with another pilot institution so that ideas and practices can be shared, developed and critiqued.

39. Paired activity in your institutional task group Using some ideas from the presentation and the publication “A Marked Improvement”, review how you might work with your colleagues back in your own institution to embed issues arising from Tenet 1 (Assessment for Learning) and Tenet 2 (Ensuring assessment is fit for purpose). Working with another institutional task group, please present your plans to date and share your ideas and any issues that have arisen. Now consider what are the opportunities and challenges that these two tenets offer and how might your plans change as a result of your conversations with colleagues here?

40. References and further reading 1 Armstrong, M., Clarkson, P. and Noble, M. (1998), Modularity and credit frameworks: the NUCCAT survey and 1998 conference report, Newcastle-upon-Tyne, Northern Universities Consortiuum for Credit Accumulation and Transfer. ASKe Weston Manor manifesto http://www.brookes.ac.uk/aske/Manifesto/ (accessed April 2012) Assessment Reform Group (1999) Assessment for Learning : Beyond the black box, Cambridge UK, University of Cambridge School of Education. Astin, A. (1997), What matters in college? Four critical years revisited. San Francisco: Jossey Bass. Bloxham, Sue. "Marking and moderation in the UK: false assumptions and wasted resources." Assessment & Evaluation in Higher Education 34.2 (2009): 209-220. Brown, G. with Bull, J. and Pendlebury, M. (1997) Assessing Student Learning in Higher Education, London: Routledge. Brown, S. and Knight, P. (1994) Assessing Learners in Higher Education, London: Kogan Page. Brown, S. Rust, C. & Gibbs, G. (1994) Strategies for Diversifying Assessment, Oxford: Oxford Centre for Staff Development. Brown, S. and Glasner, A. (eds.) (1999) Assessment Matters in Higher Education, Choosing and Using Diverse Approaches, Maidenhead: Open University Press. Brown, S. and Race, P. (2012) Using effective assessment to promote learning in Hunt, L. and Chambers, D. (2012) University Teaching in Focus, Melbourne: Australian Council for Educational Research( Acer Press). P74-91

41. References and further reading 2 Brown, S. (2011) First class: how assessment can enhance student learning in Blue Skies: new thinking about the future of higher education, London: Pearson. Boud, D. (1995) Enhancing learning through self-assessment, London: Routledge. Burgess, Robert. Beyond the honours degree classification: Burgess Group final report. Universities UK, 2007. Elander, J. and Hardman, D. (2002), ‘An application of judgement analysis to examination marking in psychology’, British Journal of Psychology 93, pp. 303-328. Elwood, J. and Klenowski, V. (2002), ‘Creating communities of shared practice: the challenges of assessment use in learning and teaching’, Assessment and Evaluation in Higher Education, 27, pp. 243-256. Gibbs, G. (1992), Improving the quality of student learning. Bristol: TES Gibbs, G (2007) ‘Departmental leadership of teaching’ Presented at The Oxford Learning Institute research seminar series, University of Oxford. Hilary term Gibbs, G. 2010, Using assessment to support student learning,Leeds Met Press, Leeds. Gibbs, P., Angelides, P. and Michaelides, P. (2004), ‘Preliminary thoughts on a praxis of higher education teaching’, Teaching in Higher Education, 9, pp. 183-194. Havnes, A. (2007), ‘What can feedback practices tell us about variation in grading across fields?’ Presented at the ASKe Seminar Series, Oxford Brookes University, 19th September. Laming, D. (1990) The reliability of a certain university examination compared with the precision of absolute judgements, Quarterly Journal of Experimental Psychology Section A—Human ExperimentalPsychology, 42 (2), pp. 239–254. Knight, P.and Yorke, M (2004),Learning, Curriculum and Employability in Higher Education. London: Routledge. McDowell, L. and Brown, S. (1998) Assessing students: cheating and plagiarism, Newcastle: Red Guide 10/11 University of Northumbria.

42. References and further reading 3 Nicol, D. J. and Macfarlane-Dick, D. (2006) Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education Vol 31(2), 199-218. Newstead, S. E. and Dennis, I. (1994), ‘Examiners examined: the reality of exam marking in psychology’, The Psychologist, 7, pp. 216-19. Nonaka, I. (1991), ‘The knowledge-creating company’, The Harvard Business Review, November-December, pp. 96-104. O’Donovan, B., Price, M. and Rust, C. (2004), ‘Know what I mean? Enhancing student understanding of assessment standards and criteria’, Teaching in Higher Education, 9, pp. 325-335. Pickford, R. and Brown, S. (2006) Assessing skills and practice, London: Routledge. Race,P., Brown, S. and Smith, B. (2005) 500 Tips on Assessment: 2nd edition, London: Routledge Polanyi, M. (1998) The tacit dimension, in: L. PRUSAK (Ed.) Knowledge in Organization (Boston, MA,ButterworthHeineman). Race, P. (2010) Making learning happen (chapter4) Sage, London Race, P. (2001) A Briefing on Self, Peer & Group Assessment, in LTSN Generic Centre Assessment Series No 9 LTSN York. Race P. (2006) The lecturer’s toolkit (3rd edition), London: Routledge. Race (2007) How to get a good degree , London, Routledge. Rust, C., Price, M. and O’Donovan, B. (2003) Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education. 28 (2), 147-164.

43. References and further reading 4 Sambell, K McDowell, L and Brown, S 1997, ‘“But is it fair?” An exploratory study of student perceptions of the consequent validity o assessment’, Studies in Educational Evaluation, vol. 23 (4), pp. 349-371. Knight, P. and Yorke, M. (2003) Assessment, learning and employability Maidenhead, UK: SRHE/Open University Press. Sadler, D. R. (1987), ‘Specifying and Promulgating Achievement Standards’, Oxford Review of Education, 13, pp. 191–209. Sadler, DR 1989, ‘Formative assessment and the design of instructional systems’, Instructional Science, vol. 18, pp. 119-144. Sadler, R. (2008) Assessment of Higher Education, in International Encyclopaedia of Education Yorke, M. (1999) Leaving Early: Undergraduate Non-completion in Higher Education, London: Routledge. Yorke, Mantz, and Bernard Longden. (2007)"The first-year experience in higher education in the UK: Report on Phase 1 of a project funded by the Higher Education Academy." Yorke, M., Bridges, P and Woolf, H. (2000), ‘Mark distributions and marking practices in UK higher education; some challenging issues’, Active Learning in Higher Education, vol. 1, no. 1, pp. 7-27.

44. Additional slides

45. Assessment Methods – is there a balance between formative and summative assessment? Every assessment method will disadvantage some students; Females tend to do better at course work; Males tend to do better with multiple choice questions; Exams in the UK provide little or no feedback. BUT: We need to plan for variety and align with our learning outcomes at both programme and module level to ensure that: • assessment methods are relevant to the 21st Century & ensure that skills, knowledge and experience are all developed. • module learning outcomes align with the programme ones. • all students can reach their full potential.

46. Some typical student comments: • All I get is a grade next to my matriculation number without any indication on how I am actually doing. • I got an essay back where the only comment was ‘use a bigger text size’, there was nothing on how to improve my grade. • For a rather lengthy scientific report, the feedback I received consisted of a mere two ticks and a question mark. • I have never received feedback for any of my exams and this means I don’t know how to improve for the third year when the marks really count.

47. Building on previous learning and developing the capacity to use feedback effectively Dialogue starters • How did this essay make you think differently? • Which of your references gave you the best information? • Which part of the assignment do you feel less confident about? • Give an example of how you could apply what you have learnt to the professional world? • What is the most important thing you have learnt from doing this assignment? • What was the most challenging part of this assignment? • What suggestions do you have for different types of assignment? (Race 2009)