Enhancing assessment and feedback: an evidence-based response

1. Enhancing assessment and feedback: an evidence-based response Berry O’Donovan & Chris Rust Deputy Directors ASKe Centre for Excellence in Teaching and Learning (Assessment Standards Knowledge exchange) Oxford Brookes University bodonovan@brookes.ac.uk, crust@brookes.ac.uk ASKe Directorate: Margaret Price, Jude Carroll, Berry O’Donovan and Chris Rust

2. Programme 10.30 Welcome and introduction 10.50 Activity 11.15 Tacit knowledge & the social-constructivist assessment process model 11.50 Student engagement with criteria - 4 case study examples 12.30 Lunch 1.30 Feedback - the issues 1.45 Activity - 7 conditions 1.55 Improving feedback - preparing students 2.05 Activity 2.20 Improving feedback - making it fit for purpose 2.40 Activity 3.10 The rest of the cycle 3.20 Summation & evaluation 3.30 End Assessment Standards Knowledge exchange

3. Three main points from the literature • Assessment and feedback are key drivers of student learning Assessment Standards Knowledge exchange

4. Assessment: a key driver of student learning “Assessment is at the heart of the student experience” (Brown, S & Knight, P., 1994) “From our students’ point of view, assessment always defines the actual curriculum” (Ramsden, P.,1992) “Assessment defines what students regard as important, how they spend their time and how they come to see themselves as students and then as graduates.........If you want to change student learning then change the methods of assessment” (Brown, G et al, 1997) Assessment Standards Knowledge exchange

5. Three main points from the literature • Assessment and feedback are key drivers of student learning • Current assessment and feedback processes are often not effective Assessment Standards Knowledge exchange

6. But there are problems… • QAA subject reviews • National Student Satisfaction Survey • “the Achilles’ heel of quality” (Knight 2002a, p. 107) • Summative assessment practices “in disarray” (Knight 2002b, p. 275 • “Broken” (Race 2003, p. 5) • “There is considerable scope for professional development in the area of assessment” (Yorke et al, 2000, p7) • Rising concern about cheating and plagiarism Assessment Standards Knowledge exchange

7. Problems contd. “The types of assessment we currently use do not promote conceptual understanding and do not encourage a deep approach to learning………Our means of assessing them seems to do little to encourage them to adopt anything other than a strategic or mechanical approach to their studies.” (Newstead 2002, p3) “…students become more interested in the mark and less interested in the subject over the course of their studies.” (Ibid, p2) Assessment Standards Knowledge exchange

8. Three main points from the literature • Assessment and feedback are key drivers of student learning • Current assessment and feedback processes are often not effective • There is lots of useful and evidence-based meta-level advice in the literature plus a plethora of articles on successful practice. Assessment Standards Knowledge exchange

9. ASKe’s focus • Communicating and engaging students with assessment requirements: standards, criteria and feedback • A key issue in assessment is that students often do not understand what is a better piece of work and do not understand what is being asked of them particularly in terms of standards and criteria (O’Donovan et al., 2001) • Feedback is also often not understood (Lea and Street, 1998), often considered too vague (Higgins, 2000) and the language complex and subject to interpretation (Lea and Street, 1998; Ridsdale 2000) Assessment Standards Knowledge exchange

10. Activity Two students are given the same written assessment task (e.g. an essay in your discipline). In the tutor feedback on the task both students are congratulated on producing a ‘highly analytical’ piece of work. One student is a postgraduate, the other a first year undergrad. Does the term ‘highly analytical’ mean something different at these two levels? And, if so, what is the difference? How do you explain to students the standard of work expected? Assessment Standards Knowledge exchange

11. Making meaning requires explicit and tacit knowledge • Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan, B., Price, M., & Rust, C., 2004) • “we can know more than we can tell” (Polanyi, reprinted 1998, p.136). • Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987) • There is a cost (in terms of time and resources) to codifying knowledge which increases the more diverse an audience’s experience and language (Snowdon, 2002). • Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991) • ‘making sense of the world’ is seen as a social and collaborative activity (Vygotsky, 1978). Assessment Standards Knowledge exchange

12. ASke’s Focus Social-constructivist view of assessment the social-constructivist view of learning argues that knowledge is shaped and evolves through increasing participation within different communities of practice the social-constructivist process model of assessment argues that students should be actively engaged with every stage of the assessment process in order that they truly understand the requirements of the process, and the criteria and standards being applied, and should subsequently produce better work(Rust C., O’Donovan, B., & Price, M., 2005) Assessment Standards Knowledge exchange

13. Active engagement with feedback Active engagement with criteria Completion and submission of work Explicit Criteria Students Assessment Standards Knowledge exchange

14. Active engagement with feedback Completion and submission of work Explicit Criteria Active engagement with criteria Students Assessment Standards Knowledge exchange

15. Marking exercise Immediate results participants av. mk non participants av. mk. Cohort 1 (99/00) 59.78 54.12 Cohort 2 (00/01) 59.86 52.86 Cohort 3 (01/02 55.7 49.7 Results 1 year later Cohort 1 57.91 51.3 Cohort 2 56.4 51.7 Rust, C., Price, M & O’Donovan, B.(2003) "Improving students’ learning by developing their understanding of assessment criteria and processes” Assessment and Evaluation in Higher Education, Vol. 28, No. 2 Assessment Standards Knowledge exchange

16. Peer marking using model answers (Forbes & Spence, 1991) Scenario: • Engineering students had weekly maths problem sheets marked and problem classes • Increased student numbers meant marking impossible and problem classes big enough to hide in • Students stopped doing problems • Exam marks declined (Average 55%>45%) Solution: • Course requirement to complete 50 problem sheets • Peer assessed at six lecture sessions but marks do not count • Exams and teaching unchanged Outcome: Exam marks increased (Av. 45%>80%) Assessment Standards Knowledge exchange

17. Peer feedback - Geography (Rust, 2001) Scenario • Geography students did two essays but no apparent improvement from one to the other despite lots of tutor time writing feedback • Increased student numbers made tutor workload impossible Solution: • Only one essay but first draft required part way through course • Students read and give each other feedback on their draft essays • Students rewrite the essay in the light of the feedback • In addition to the final draft, students also submit a summary of how the 2nd draft has been altered from the1st in the light of the feedback Outcome: Much better essays Assessment Standards Knowledge exchange

18. Peer feedback - Computing (Zeller, 2000*) The Praktomat system allows students to read, review, and assess each other’s programs in order to improve quality and style. After a successful submission, the student can retrieve and review a program of some fellow student selected by Praktomat. After the review is complete, the student may obtain reviews and re-submit improved versions of his program. The reviewing process is independent of grading; the risk of plagiarism is narrowed by personalized assignments and automatic testing of submitted programs. In a survey, more than two thirds of the students affirmed that reading each other’s programs improved their program quality; this is also confirmed by statistical data. An evaluation shows that program readability improved significantly for students that had written or received reviews. [*Available at: http://www.infosun.fim.unipassau.de/st/papers/iticse2000/iticse2000.pdf] Assessment Standards Knowledge exchange

19. Active engagement with criteria Completion and submission of work Explicit Criteria Active engagement with feedback Students Assessment Standards Knowledge exchange

20. Potential of feedback Feedback is the most powerful single influence that makes a difference to student achievement Hattie (1987) - in a comprehensive review of 87 meta-analyses of studies Feedback has extraordinarily large and consistently positive effects on learning compared with other aspects of teaching or other interventions designed to improve learning Black and Wiliam(1998) - in a comprehensive review of formative assessment Students are hungry for feedback to develop their learning (Higgins et al, 2002) Assessment Standards Knowledge exchange

21. Retention and feedback The number of opportunities available for feedback is an important variable in non-completion of students in the early years of study (Yorke, 1999) Where students are uncertain about their ability to succeed, formative feedback is of particular importance (Yorke & Longden, 2004) Assessment Standards Knowledge exchange

22. Feedback problems • Unhelpful feedback (Maclellan, 2001) • Too vague (Higgins, 2000) • Subject to interpretation (Ridsdale, 2003) • Not understood (e.g. Lea and Street, 1998) • Don’t read it (Hounsell, 1987) • Damage self-efficacy (Wotjas, 1998) • Has no effect (Fritz et al, 2000) • Seen to be too subjective (Holmes & Smith, 2003) Assessment Standards Knowledge exchange

23. 11 conditions under which assessment supports learning 1 (Gibbs and Simpson, 2002) • Sufficient assessed tasks are provided for students to capture sufficient study time (motivation) • These tasks are engaged with by students, orienting them to allocate appropriate amounts of time and effort to the most important aspects of the course (motivation) • Tackling the assessed task engages students in productive learning activity of an appropriate kind (learning activity) • Assessment communicates clear and high expectations (motivation) Assessment Standards Knowledge exchange

24. 11 conditions under which assessment supports learning 2 (Gibbs and Simpson, 2002) 5 Sufficient feedback is provided, both often enough and in enough detail 6 The feedback focuses on students’ performance, on their learning and on actions under the students’ control, rather than on the students themselves and on their characteristics 7 The feedback is timely in that it is received by students while it still matters to them and in time for them to pay attention to further learning or receive further assistance 8 Feedback is appropriate to the purpose of the assignment and to its criteria for success. 9 Feedback is appropriate, in relation to students’ understanding of what they are supposed to be doing. 10 Feedback is received and attended to. 11 Feedback is acted upon by the student Assessment Standards Knowledge exchange

25. Improving feedback - prepare students (in Yr 1 esp.) • Aligning expectations (of staff & students, & between teams of markers) - often a mismatch of expectations e.g correcting errors, advice for the future, diagnosis of general problems, comments specific only to that piece of work. These mismatches occur frequently with no particular pattern about who holds which view/perspective but problems arise when the the two don't coincide. Purpose of feedback may vary from assignment to assignment so would need to be clarified each time. (Freeman & Lewis, 1998) • Identifying all feedback available • Model the application of feedback - e.g. using previously-marked assignments to show how feedback was used to improve later assignments • Encourage the application of feedback - e.g. in a subsequent piece of work the student is required to show how they have used prior feedback to try to improve their work and some marks allocated for this. • Require and develop self-assessment it is the interaction between both believing in self-responsibility and using assessment formatively that leads to greater educational achievements (Brown & Hirschfeld, 2008) Assessment Standards Knowledge exchange

26. Activity In 3’s, discuss: • How do you currently prepare students to understand and engage with feedback? • Which of these ideas could you introduce, or develop further, and how? Assessment Standards Knowledge exchange

27. Improving feedback - ensuring engagement • Ensure students have MOM - Motive, Opportunity, Means (Angelo, 2007) • Draft-plus-rework - feedback effort (for markers and students) is located at the draft stage, and possibly only a summative grade is given for the final submission • Improve the linkage of assessment strategies across programmes and between modules/units • Increase student engagement and understanding through dialogue - in-class discussion of exemplars, peer-review discussions supported by tutors, learning-sets, etc. • Identify what is feasible in a given assessment context - written feedback can often do little more than ‘diagnose’ development issues and then direct students to other resources for help and support • Ensure it is timely - ‘quick and dirty’ generic feedback, feedback on a draft, MCQs & quizzes, etc. (using technology may help) • Consider the role of marks - they obscure feedback • Reduce over-emphasis on written feedback - oral can be more effective (McCune, 2004). But individual F2F can be resource intensive • Review resource allocations (N.B. OU 60%) Assessment Standards Knowledge exchange

28. Figure 1: Peer-review as a method of encouraging students to discuss and compare their understanding of assessment criteria Assessment Standards Knowledge exchange

29. Figure 2: the use of 'exemplars' as amechanism for encouraging dialogue about assessment criteria Assessment Standards Knowledge exchange

30. Figure 3: Generic feedback and self critique Assessment Standards Knowledge exchange

31. Activity Individually: Choose one or more specific ideas to improve feedback that you think you could use. In as much detail as possible, identify how you would put the idea/s into practice. In pairs: Take it in turns to explain your plans to your partner. The job for the listener is to be a friendly and constructive critic Assessment Standards Knowledge exchange

32. 7 principles of good feedback practice (Nicol and Macfarlane-Dick, 2006) Good feedback practice: • helps clarify what good performance is (goals, criteria, expected standards) • facilitates the development of reflection and self-assessment in learning • delivers high-quality information to students about their learning • encourages teacher and peer dialogue around learning • encourages positive motivational beliefs and self-esteem • provides opportunities to close the gap between current and desired performance • provides information to teachers that can be used to help shape the teaching Assessment Standards Knowledge exchange

33. Assessment design & development of explicit criteria Active engagement with feedback Active engagement with criteria Completion and submission of work Tutor discussion of criteria Staff Assessment guidance to staff Marking and moderation Explicit Criteria Students Assessment Standards Knowledge exchange

34. Assessment design & development of explicit criteria Active engagement with feedback Active engagement with criteria Completion and submission of work Rust C.,O’Donovan B & Price., M (2005) Tutor discussion of criteria Staff Assessment guidance to staff Marking and moderation Explicit Criteria Students Assessment Standards Knowledge exchange