Assessment as Learning Experience '

1. Business School Assessment as Learning Experience ' Prof Margaret Price ASKe Centre for Excellence

2. Outline • The problem with assessment • Where should we go from here? • Clarity about purpose and assessment standards • Reliability and the nature of assessment standards • Engaging students in the assessment process. • Assuring standards • Implications Business School ASKe – Assessment Standards Knowledge exchange

3. We are not doing well. • QAA reviews • National Student Satisfaction Survey • “the Achilles’ heel of quality” (Knight 2002a, p. 107) • Summative assessment practices “in disarray” (Knight 2002b, p. 275 • “Broken” (Race 2003, p. 5) • “There is considerable scope for professional development in the area of assessment” (Yorke et al, 2000, p7) Business School ASKe – Assessment Standards Knowledge exchange

4. Assessment Business School ASKe – Assessment Standards Knowledge exchange

5. Business School ASKe – Assessment Standards Knowledge exchange

6. Where should we go from here? • Weston Manor Group, November 07 (40+ National and International Experts in Assessment) • Two days of discussions Outcome: Assessment Standards: A Manifesto for Change • Six tenet manifesto for change to assessment practice related to standards Business School ASKe – Assessment Standards Knowledge exchange

7. Assessment: a key driver of student learning “Assessment is at the heart of the student experience” (Brown, S & Knight, P., 1994) “From our students’ point of view, assessment always defines the actual curriculum” (Ramsden, P.,1992) “Assessment defines what students regard as important, how they spend their time and how they come to see themselves as students and then as graduates.........If you want to change student learning then change the methods of assessment” (Brown, G et al, 1997) Business School ASKe – Assessment Standards Knowledge exchange

8. Assessment standards in HE “The types of assessment we currently use do not promote conceptual understanding and do not encourage a deep approach to learning………Our means of assessing them seems to do little to encourage them to adopt anything other than a strategic or mechanical approach to their studies.” (Newstead 2002, p3) “Conventional assessment procedures are unable to do justice to the most important outcomes of any educational process worth the name” (Raven 1991, p1) Business School ASKe – Assessment Standards Knowledge exchange

9. Tenet 1 “The debate on standards needs to focus on how high standards of learning can be achieved through assessment. This requires a greater emphasis on assessment for learning rather than assessment of learning.” Business School ASKe – Assessment Standards Knowledge exchange

10. The problems with reliability ‘This quest for reliability tends to skew assessment towards the assessment of simple and unambiguous achievements, and considerations of cost add to the skew away from judgements of complex learning’ (Knight 2002b p278) “…summative judgement itself is the problem” (Burgess, 2007, p. 8) Many research findings indicate a declining use of deep and contextual approaches to study as students’ progress through their degree programmes (Watkins & Hattie, 1985; Kember et al, 1997; Richardson, 2000; Zhang & Watkins, 2001) Business School ASKe – Assessment Standards Knowledge exchange

11. “…students become more interested in the mark and less interested in the subject over the course of their studies.” (Newstead 2002, p2) “Even when lecturers say that they want students to be creative and thoughtful, students often recognise that what is really necessary, or at least what is sufficient, is to memorise” (Gibbs, 1992, p. 10) Our current systems focused on marks and grades aren’t working (Elander & Hardman, 2002; Laming, 1990; Yorke et al, 2002; Bridges et al, 2002; Newstead and Dennis, 1994) Business School ASKe – Assessment Standards Knowledge exchange

12. Tenet 2 “When it comes to the assessment of learning, we need to move beyond systems focused on marks and grades towards the valid assessment of the achievement of intended programme outcomes.” Business School ASKe – Assessment Standards Knowledge exchange

13. Reliability and Standards • Defining standards • Clarity • Transparency • How do students come to understand standards? Business School ASKe – Assessment Standards Knowledge exchange

14. Active student engagement 3. The Social Constructivist Model 4. The ‘Cultivated’ Community of Practice Model The Future Formal activities and inputs Informal activities and inputs The Past 1. The Traditional Model – 2. The ‘Dominant Logic’ Explicit Model Passive student engagement O’Donovan, Price & Rust 2008 Business School ASKe – Assessment Standards Knowledge exchange

15. Active student engagement 3. The Social Constructivist Model • 4. The ‘Cultivated’ Community of Practice Model • . The Future Formal activities and inputs Informal activities and inputs The Past 1. The Traditional Model –Standards absorbed over relatively longer times informally and serendipitously 2. The ‘Dominant Logic’ Explicit Model Passive student engagement O’Donovan, Price and Rust. 2008 Business School ASKe – Assessment Standards Knowledge exchange

16. Active student engagement 3. The Social Constructivist Model • 4. The ‘Cultivated’ Community of Practice Model The Future Formal activities and inputs Informal activities and inputs The Past 1. The Traditional Model –Standards absorbed over relatively longer times informally and serendipitously 2. The ‘Dominant Logic’ Explicit Model Standardsexplicitlyarticulated (with limitations) and passively presented to students Passive student engagement O’Donovan, Price and Rust. 2008 Business School ASKe – Assessment Standards Knowledge exchange

17. Limitations of explicit articulation • Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004) • “we can know more than we can tell” (Polanyi, reprinted 1998, p.136). • Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987) • There is a cost (in terms of time and resources) to codifying knowledge which increases the more diverse an audience’s experience and language (Snowdon, 2002). • Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991) Business School ASKe – Assessment Standards Knowledge exchange

18. Nature of criteria • An indispensable condition for improvement in student learning is that “the student comes to hold a concept of quality roughly similar to that held by the teacher” (Sadler, 1989) • Regulative and logical criteria “standards can be defined in terms of well-defined outcomes” (Sadler, 1987, p. 70) Business School ASKe – Assessment Standards Knowledge exchange

19. Nature of Criteria (2) • Prescriptive and constitutive criteria refer to matters of degree “It would be difficult or impossible to guess the educational level at which they are applicable… ” (Sadler, 1987, p. 70). • Such types of criteria are often interdependent and can only be assessed using holistic/professional judgement (Sadler, 2008) • Such criteria are socially constructed requiring the sharing of tacit knowledge over time (O’Donovan et al, 2004; Rust et al, 2005) Business School ASKe – Assessment Standards Knowledge exchange

20. Tenet 3 “ Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently, diminish the learning experience and threaten its validity. There are important benefits of higher education which are not amenable either to the precise specification of standards or to objective assessment.” Business School ASKe – Assessment Standards Knowledge exchange

21. Active student engagement 3. The Social Constructivist Model Actively engaging students in formal processes to communicate tacit knowledge of standards • 4. The ‘Cultivated’ Community of Practice Model The Future Formal activities and inputs Informal activities and inputs The Past 1. The Traditional Model Standards absorbed over relatively longer times informally and serendipitously 2. The ‘Dominant Logic’ Explicit Model Standardsexplicitlyarticulated (with limitations) and passively presented to students Passive student engagement O’Donovan, Price and Rust. 2008 Business School ASKe – Assessment Standards Knowledge exchange

22. Understanding assessment standards • ‘making sense of the world’ is a social and collaborative activity (Vygotsky, 1978). • Passive receipt of feedback has little effect on future performance (Fritz, et al., 2000) • ASKe assessment intervention (Rust et al, 2003;1,2,3 leaflet) • Dialogue and participatory relationships are key elements of engaging students with assessment feedback (Price et al 2008) Business School ASKe – Assessment Standards Knowledge exchange

23. What works? • The most significant factor in student academic success is student involvement fostered by student/staff interactions and student/student interactions (Astin, 1997) • The only common factor in a study of departments deemed excellent in both research and learning and teaching is high levels of student involvement (Gibbs, 2007) • “participation, as a way of learning, enables the student to both absorb, and be absorbed in the culture of practice” (Elwood & Klenowski, 2002, p. 246) Business School ASKe – Assessment Standards Knowledge exchange

24. Active student engagement 3. The Social Constructivist Model Actively engaging students in formal processes to communicate tacit knowledge of standards • 4. The ‘Cultivated’ Community of Practice Model • Tacit standards communicated through participation in informal knowledge exchange networks ‘seeded’ by specific activities. The Future Formal activities and inputs Informal activities and inputs The Past 1. The Traditional Model Tacit standards absorbed over relatively longer times informally and serendipitously 2. The ‘Dominant Logic’ Explicit Model Standardsexplicitlyarticulated (with limitations) and passively presented to students Passive student engagement O’Donovan, Price and Rust. 2008 Business School ASKe – Assessment Standards Knowledge exchange

25. Tenet 4 “Assessment standards are socially constructed so there must be a greater emphasis on assessment and feedback processes that actively engage both staff and students in dialogue about standards. It is when learners share an understanding of academic and professional standards in an atmosphere of mutual trust that learning works best.” Business School ASKe – Assessment Standards Knowledge exchange

26. Coming to know • Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke 2004) • The achievement of high-level learning requires integrated and coherent progression based on programme outcomes • Where there is a greater sense of the holistic programme students are likely to achieve higher standards than on more fragmented programmes (Havnes, p. 2007) • Students need to engage as interactive partners in a learning community, relinquishing the passive role of ‘the instructed’ within processes controlled by academic experts (Gibbs et al, 2004) Business School ASKe – Assessment Standards Knowledge exchange

27. Tenet 5 “ Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of standards and monitor and supervise their own learning.” Business School ASKe – Assessment Standards Knowledge exchange

28. Community and professional judgement • Changes in higher education (e.g. massification, reduced unit of resource, expectations of increased productivity in staff) threaten the ‘health’ of disciplinary communities and their ability to share and exemplify professional judgement. • There has been slow progress in the professionalisation of university teachers • There has been limited attention paid to professional assessment practice • Professional judgement must be within a framework of professionalism Business School ASKe – Assessment Standards Knowledge exchange

29. Tenet 6 “ Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities.” Business School ASKe – Assessment Standards Knowledge exchange

30. Key points • The current dominant logic of the explicit approach to assessment standards is insufficient and was not defended by any of the 40 experts gathered at Weston Manor • The quest for reliability is getting in the way of learning • Our main concern should be enabling students to achieve high level, complex learning. • Active involvement by staff and students in the learning community is essential to reach common understandings of assessment standards • To achieve change it is likely that a review and evaluation of the allocation of time and resources within HE will be needed. Business School ASKe – Assessment Standards Knowledge exchange

31. What next? • If you would like to be a signatory to this manifesto please visit http://www.business.brookes.ac.uk/learningandteaching/aske/ Business School ASKe – Assessment Standards Knowledge exchange

32. References • Astin, A. (1997) What matters in college? Four critical years revisited. San Francisco: Jossey Bass • Bridges, P., Cooper, A., Evanson, P., Haines, C., Jenkins, D., Scurry, D., Woolf, H. and Yorke, M (2002), ‘Coursework marks high examination marks low: discuss’, Assessment and Evaluation in Higher Education, vol. 27, no. 1, pp. 35-48. • Brown, G., Bull, J. & Pendlebury, M. (1997) Assessing student learning in higher education. London: Routledge • Brown, S & Knight, P. (1994) Assessing Learners in Higher Education. London: Kogan Page • Burgess, R. (2007) Beyond the Honours Degree Classification: Burgess Group Final Report. Universities UK • Elander, J. and Hardman, D. (2002), ‘An application of judgement analysis to examination marking in psychology’, British Journal of Psychology 93, pp. 303-328. • Elwood, J. & Klenowski, V. (2002) ‘Creating communities of shared practice: the challenges of assessment use in learning and teaching’, Assessment and Evaluation in Higher Education, 27, 243-256 • Fritz, C.O., Morris, P.E., Bjork, R.A., Gelman, R. & Wickens, T.D. (2000) ‘When further learning fails: stability and change following repeated presentation of text’, British Journal of Psychology, 91, 493-511 • Gibbs, G (2007) Departmental leadership of teaching Presented at Learning Institute, University of Oxford 8 Feb • Gibbs, G. (1992) Improving the quality of student learning. Bristol: TES • Gibbs, P., Angelides, P. and Michaelides, P. (2004) ‘Preliminary thoughts on a praxis of higher education teaching’, Teaching in Higher Education, 9, 183-194 • Havnes, A. (2007) ‘What can feedback practices tell us about variation in grading across fields?’ Presented at the ASKe Seminar Series, Oxford BrookesUniversity, 19th September Business School ASKe – Assessment Standards Knowledge exchange

33. Kember, D., et al. (1997) Case studies in improving teaching and learning from the action learning project. Action learning project, Hong Kong • Knight, P. T. (2002a) The Achilles’ heel of quality: the assessment of student learning, Quality in Higher Education, 8(1), 107–115. • Knight, P. T. (2002b) ‘Summative assessment in higher education: practices in disarray’, Studies in Higher Education, 27(3), 275–286 • Knight, P and Yorke, M (2004) Learning, Curriculum and Employability in Higher Education. London: Routledge • Laming, D (1990) The reliability of a certain University examination compared with the precision of absolute judgements, Quarterly Journal of Experimental Psychology Section A - Human Experimental Psychology, Vol. 42 No.2, 239-254 • Newstead, S. (2002) ‘Examining the examiners: why are we so bad at assessing students?’, Psychology Learning and Teaching, 2 (2), • Newstead, S. E. and Dennis, I. (1994) ‘Examiners examined: the reality of exam marking in psychology’, The Psychologist, 7, 216-19 • Nonaka, I.. (1991) ‘The knowledge-creating company’, The Harvard Business Review, November-December, 96-104 • O’Donovan, B., Price, M. & Rust, C. (2004) ‘Know what I mean? Enhancing student understanding of assessment standards and criteria’, Teaching in Higher Education, 9, 325-335 • O’Donovan, B.,Price, M & Rust, C. (2008) Developing student understanding of assessment standards: a nested hierarchy of approaches. Teaching in Higher Education13(2), 205-218 • Polanyi, M. (1998) The tacit dimension, Reprinted in L. Prusak (Ed) Knowledge in Organizations (Boston, Butterworth Heineman). • Price, M., K. Handley, & B. O’Donovan. (2008). Feedback – all that effort but what is the effect?, Paper presented at the EARLI/Northumbria Assessment Conference. August 27-29 in Potsdam, Germany. Business School ASKe – Assessment Standards Knowledge exchange

34. Race, P. (2003) Why fix assessment? in: L. Cooke & P. Smith (Eds) Seminar: reflections on learning and teaching in higher education, High Wycombe: Buckinghamshire Chilterns University College. • Ramsden, P. (1992) Learning to teach in higher education. London: Routledge • Raven, J. (1991) The tragic illusion: educational testing. New York: Trillium Press • Rust, C, Price, M. & O’Donovan, B. (2003) Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation. 28, 147-164. • Rust, C., O’Donovan, B. & Price, M. (2005) ‘A social constructivist assessment process model: how the research literature shows us this could be best practice’, Assessment and Evaluation in Higher Education, 30 (3), 231-240 • Sadler, D. R. (1987) ‘Specifying and Promulgating Achievement Standards’, Oxford Review of Education, 13, 191–209 • Sadler, D. R. (1989) ‘Formative assessment and the design of instructional systems’, Instructional Science, 18, 119-144 • Sadler, D. R. (2008) ‘Indeterminacy in the use of preset criteria for assessment and grading’, Assessment and Evaluation in Higher Education, April • Snowdon, D (2002) Complex acts of knowing: paradox and descriptive self-awareness. Journal of Knowledge Management, 6(Special Edition), pp. 100-111. • Watkins, D., and Hattie, J. (1985) ‘A longitudinal study of the approaches to learning of Australian tertiary students’, Human Learning, 4, 127-41 • Vygotsky, L. S. (1978) Mind in society: the development of higher psychological processes. MA: Harvard University Press • Yorke, M., Bridges, P. & Woolf, H. (2000) Mark distributions and marking practices in UK higher education; some challenging issues, Active Learning in Higher Education, 1 (1), 7-27 • Yorke, M. (2002) Subject Benchmarking and the assessment of student learning, Quality Assurance in Education, 10(3), 155-171. • Zhang, L. F. & Watkins, D. (2001) ‘Cognitive development and student approaches to learning: an investigation of Perry's theory with Chinese and US university students’, Higher Education, 41, 236-261 Business School ASKe – Assessment Standards Knowledge exchange