Assessment matters University of Leiden 30 November 2012

1. Assessment mattersUniversity of Leiden30 November 2012 Sally Brown http://sally-brown.net Emerita Professor, Leeds Metropolitan University, Adjunct Professor, University of the Sunshine Coast, Central Queensland and James Cook University Queensland Visiting Professor, University of Plymouth and Liverpool John Moores University & Honorary Fellow, University of Northumbria

2. 1. How to lead assessment for learning in universities Leaders can impact on the assessment context by • Reviewing student experiences of assessment and feedback, seeking opportunities for enhancement; • Establishing some clear and consistent ground rules (for example, that assessed work must be returned within 3 weeks working for continuing students); • Monitoring compliance with ground rules and following up when good practice is not being achieved; • Providing opportunities for colleagues to share their own good practice together with staff development on innovations.

3. Why is assessment such a big issue? Good feedback and assessment practices are essential to student learning; Student satisfaction surveys frequently highlight significant dissatisfaction around these issues; In tough times, staff often find the pressure of achieving fast and formative feedback a heavy chore.

4. To integrate assessment we need to realign it with the curriculum by: • Exploring ways in which assessment can be made integral to learning. • Constructively aligning (Biggs, 2003) assignments with planned learning outcomes and the curriculum taught; • Providing realistic tasks: students are likely to put more energy into and play fairer with assignments they see as authentic and worth bothering with.

5. Strategies for ensuring assessment is for rather than of learning • It needs to be built-in rather than bolt-on; • Assignments need to be authentic, that is, assessing learning that is identified in the learning outcomes; • Learning outcomes need to be designed to be specific, measurable, achievable, realistic and time-constrained (SMART); • The assessment strategy should make sure that assignments are fit-for-purpose.

6. Twelve questions on assessment. You can: • Use these to help to design an authentic assessment approach at course design stage; • Use them also as an aid to curriculum refreshment activity and prior to periodic review.

7. Assessment for learning: is assessment fully integrated within learning activities or is it an add-on that adds nothing to student engagement? • Preparation: are you developing students’ assessment literacy, so they understand fully what is required of them and can optimise their performances in a range of assessment contexts? Are staff inducted so they all share understandings of assessment practice. • Purpose: are you clear about why on each occasion you are assessing? Is it to give students guidance on how to improve or remediate work, or it is a scoring exercise to determine final grades? Is it focussing on theory or practice (or an integration of the two)?

8. Pacing and timing: are you offering feedback and assessment opportunities throughout the learning period or are assignments bunched together (particularly right at the end of the module)? Are you ensuring that students don’t have multiple assignments from different modules with the same submission date? • Volume of assessment: are you offering sufficient opportunities for students to learn through assessment without exhausting staff and putting excessive pressure on students in terms of workload? • Constructive alignment: is it clear how the assignments link to the learning outcomes, and do you offer good coverage of subject material and capabilities (or are you encouraging guessing of topics and risk taking activities)?

9. Variety: are you enabling students to demonstrate capability in diverse ways or are you reusing the same methods (essays, reports, unseen time-constrained exams) over and over again? • Inclusivity: Are students’ special needs in terms of assessment designed into assignments from the outset or do you have to make special arrangements for students with dyslexia, visual or aural impairments or other disabilities responsively rather than proactively? • Agency: is all your assessment undertaken by tutors or do you also use peers, students themselves, employers and clients?

10. Feedback: how fast can you provide it and what assurances can you give to students about its usefulness and ability to feed into future assignments? • Quality assurance: are you able to demonstrate that your assessment is fair, consistent and reliable? Will external scrutineers recognise the integrity of the assessment process? • Technology: are you using computer aided assessment where it is most useful (for drills and checking learning) enabling assessor time to be used most effectively where judgment is required?

11. Good feedback practice: 1. Helps clarify what good performance is (goals, criteria, expected standards); 2. Facilitates the development of self-assessment (reflection) in learning; 3. Delivers high quality information to students about their learning; 4. Encourages teacher and peer dialogue around learning; 5. Encourages positive motivational beliefs and self-esteem; 6. Provides opportunities to close the gap between current and desired performance; 7. Provides information to teachers that can be used to help shape the teaching. (Nicol and McFarlane Dick)

12. Encouraging students to take assessment more seriously • All assessment needs to be seen to be fair, consistent, reliable, valid and manageable; • Many assessment systems fail to clarify for students the purposes of different kinds of assessment activity; • Low-stakes early formative assessment helps students, especially those from disadvantaged backgrounds, understand the rules of the game.

13. Making assessment work well • Intra-tutor and Inter-tutor reliability need to be assured; • Practices and processes need to be transparently fair to all students; • Cheat and plagiarisers need to be deterred/punished; • Assessment needs to be manageable for both staff and students; • Assignments should assess what has been taught/learned not what it is easy to assess.

14. Can we provide opportunities for multiple assessment? • Consider allowing resubmissions of work as part of a planned programme; • Students often feel they could do better once they have seen the formative feedback and would like the chance to have another go; • Particularly at the early stages of a programme, we can consider offering them the chance to use formative feedback productively; • Feedback often involves a change of orientation, not just the remediation of errors.

15. Using formative assessment to promote independence and learning • Investigate how learning can be advanced in small steps using a ‘scaffolding’ approach; • Provide lots of support in the early stages when students don’t understand the ‘rules of the game’ and may lack confidence; • This can then be progressively removed as students become more confident in their own abilities.

16. Play fair with students by avoiding using ‘final language’ (Boud) • Avoid destructive criticism of the person rather than the work being assessed. • Try not to use language that is judgmental to the point of leaving students nowhere to go. • Words like “appalling”, “disastrous” and “incompetent” give students no room to manoeuvre. • However, words like ”incomparable” and “unimprovable” don’t help outstanding students to develop ipsatively either.

17. Play fair by giving feedback to students with diverse abilities • Students at the top end of the ability range sometimes feel short changed by minimal feedback; • Students with many weaknesses easily become dispirited if there is too much negative feedback; • Consider giving an assessment sandwich. Start with something positive, go into the detailed critique and find something nice to say at the end (to motivate them to keep reading!); • Explore ways to incentivise reading of feedback; • Consider which medium to use for students with disabilities (e.g. don’t use bad handwriting for those with visual impairments or dyslexia!).

18. Conclusions to part one • Assessment impacts highly on student learning so we need to rethink how we can best do this, taking account of new contexts, new technologies and new opportunities; • Efficient and effective feedback is just about the most important thing we do to enhance student learning, progression and success.

19. Part two: A case study by Erik de Maaker, who uses audiovisual material together with a thesis at the end of a Master's programme.

20. Part 3: Differentiating Masters level and undergraduate level assessment

21. The context for reviewing M-level assessment • Most HEIs are aiming to increase the number of post-graduate students they recruit; • In many nations, undergraduate recruitment is at, or close to saturation; • In a competitive global environment, Masters programmes need to have a competitive edge; • Authentic assessment can be a Unique Selling Point for Masters Programmes.

22. At Masters level, assessment really matters! • Many Masters programmes are professionally-orientated or vocational hence the need for a strong focus on authentic assessment; • Students have high levels of expectation from their tutors at Masters level; • Most M-level programmes are assessed very conservatively, using written assignments including dissertations, unseen time constrained exams and essays; • We need to distinguish our programmes from those offered by our competitors worldwide.

23. Higher education providers may offer a Master's degree with the specific intention of: • Enabling students to focus on a particular aspect of a broader subject area in which they have prior knowledge or experience through previous study or employment; and/or • Enabling students to focus on a particular subject area or field of study in greater depth than they encountered during the course of previous study or experience. This may include enabling students to develop knowledge of a new discipline or field of study in combination with a relevant subject area in which they have prior knowledge or experience; and/or • Enabling students to learn how to conduct research, often linked to a particular discipline or field of study.

24. QAA in Scotland: guidance on level 11 qualifications (I like this) Characteristic outcomes of Masters degrees i A systematic understanding of knowledge, and a critical awareness of current problems and/or new insights, much of which is at, or informed by, the forefront of their academic discipline, field of study, or area of professional practice. ii A comprehensive understanding of techniques applicable to their own research or advanced scholarship. iii Originality in the application of knowledge, together with a practical understanding of how established techniques of research and enquiry are used to create and interpret knowledge in the discipline. iv Conceptual understanding that enables the student: • to evaluate critically current research and advanced scholarship in the discipline; and • to evaluate methodologies and develop critiques of them and, where appropriate, to propose new hypotheses.

25. Typically, holders of the qualification will be able to: • deal with complex issues both systematically and creatively, make sound judgements in the absence of complete data, and communicate their conclusions clearly to specialist and non-specialist audiences; • demonstrate self-direction and originality in tackling and solving problems, and act autonomously in planning and implementing tasks at a professional or equivalent level; • continue to advance their knowledge and understanding, and develop new skills to a high level; and will have: • the qualities and transferable skills necessary for employment requiring: (i) the exercise of initiative and personal responsibility; (ii) decision-making in complex and unpredictable situations; and (iii) the independent learning ability required for continuing professional development.

26. My questions: mapping the student experience at Master’s Level • Will students feel from the outset that they are on a Master’s programme? • Are you ensuring that students are immersed in the subject they have come to study from the outset? • Is induction a valuable and productive introduction to the course (or just the distribution of bags and bags of paper)? • Do students have a positive and balanced experience across the programme? • Are there points in the academic year when there doesn’t seem to be much going on?

27. Assimilate has been a 3-year NTFS funded project • We’ve been exploring innovative assessment at Masters level using research funding from the National Teaching Fellowship scheme. • Recognising that limited prior research had been undertaken in this area, we’ve been reviewing assessment methods used to assess at this level, particularly exploring authentic assessment. • Interviews were undertaken in the UK and internationally by students and team members to elicit information about diverse approaches and to produce case studies showcasing innovations.

28. Emergent outcomes • Using data from 45 interviews undertaken by the project team, and from conversations with other people we’ve encountered who have an interest in the area, we’ve produced 44 case studies illustrating diverse M-level assessment together with seven vignettes and three national overviews as included in the Compendium; • We have been impressed by the diversity and creativity of many of the approaches adopted; • We’ve also encountered much traditional written assessment, especially dissertations, unseen time-constrained exams, essays and project reports.

29. Good practice M-level Assessment examples include: • Highly authentic assignments, which relate closely to programme outcomes; • Multiple assessments which build incrementally to final submission; • Good feedback opportunities, giving students the chance to benefit from advice to improve performance; • Assignments that require teamwork and group activity; • Assignments that foster employability and that foster employer engagement; • Assignments that are enhanced and supported by technology; • Assignements requiring peer engagement / peerassessment.

30. Other learning points • It’s been interesting to observe how fuzzy are common understandings of the differences between M-level and undergraduate level assessment; • The importance of authentic assessment to professionally-orientated Masters programmes has been highlighted; • We’ve learned about variations in practice at M-level between different national systems, especially in terms of duration of programmes and funding arrangements; • We’ve also developed as individuals and as members of the project team, learning particularly about project management.

31. Selected references and further reading (1) Biggs J (2003) Teaching for Quality Learning at University (Maidenhead: SRHE & Open University Press) Bowl, M (2003) Non-traditional entrants to higher education ‘they talk about people like me’ Stoke on Trent, UK, Trentham Books Brown, S. Rust, C & Gibbs, G (1994) Strategies for Diversifying Assessment Oxford Centre for Staff Development. Boud, D. (1995) Enhancing learning through self-assessment London: Routledge. Brown, S., Race, P. and Bull, J. (eds.) (1999) Computer Assisted Assessment in Higher Education London: Routledge. Brown, G. with Bull, J. and Pendlebury, M. (1997) Assessing Student Learning in Higher Education London: Routledge. Brown, S. (2012) ‘What are the perceived differences between assessing at Masters level and undergraduate level assessment? Some findings from an NTFS–funded project’ Innovations in Education and Teaching International, forthcoming

32. Useful references: 2 Brown, S., Deignan, T., Race, P. and Priestley, J. (2012) Assessing students at Masters Level: learning points for Educational Developers, Educational Developments, SEDA, Birmingham. Brown, S. (2012) Diverse and innovative assessment at Masters Level: alternatives to conventional written assignments, in AISHE-J: The All Ireland Journal of Teaching and Learning in Higher Education Vol 4, No 2. Brown, S. and Glasner, A. (eds.) (1999) Assessment Matters in Higher Education, Choosing and Using Diverse Approaches, Maidenhead: Open University Press. Brown, S. and Knight, P. (1994) Assessing Learners in Higher Education, London: Kogan Page.

33. Useful references 3 Carroll, J. and Ryan, J. (2005) Teaching International students: improving learning for all. London: Routledge SEDA series. Falchikov, N, (2004) Improving Assessment through Student Involvement: Practical Solutions for Aiding Learning in Higher and Further Education, London: Routledge. Gibbs, G. (1999) Using assessment strategically to change the way students learn, In Brown, S. & Glasner, A. (eds.), Assessment Matters in Higher Education: Choosing and Using Diverse Approaches, Maidenhead: SRHE/Open University Press. Hunt, L. and Chalmers, D. (2012) University Teaching in Focus: a learning-centred approach, Abingdon: Routledge .

34. Useful references 4 Institute of Education (2006) Masters level criteria for Geography PGCE http://www.geography.org.uk/download/GA_PRGTIPBrooksMLevelCriteria.pdf Accessed March 2012 Kneale, P. E. (1997) The rise of the "strategic student": how can we adapt to cope? in Armstrong, S., Thompson, G. and Brown, S. (eds) Facing up to Radical Changes in Universities and Colleges, 119-139 London: Kogan Page. Knight, P. and Yorke, M. (2003) Assessment, learning and employability, Maidenhead, UK: SRHE/Open University Press. McDowell, E. & Brown, S. (1998) Assessing students: cheating and plagiarism, Red Guide 10/11 University of Northumbria, Newcastle Nicol, D. J. and Macfarlane-Dick, D. (2006) Formative assessment and self-regulated learning: A modeland seven principles of good feedback practice,Studies in Higher Education (2006), Vol.31(2), 199-218.

35. Useful references 5 Price, M., Rust, C., Donovan, B., and Handley, K. with Bryant, R. (2012) Assessment Literacy: the foundation for improving student learning, Oxford: Oxford Centre for Staff and learning Development. Sadler, D. R. (1989) Formative assessment and the design of instructional systems Instructional Science 18, 119-144. Sadler, D. R. (1998) Formative assessment: revisiting the territory Assessment in Education: Principles, Policy and Practice 5, 77-84. Pickford, R. and Brown, S. (2006) Assessing skills and practice London: Routledge. QAA (2010) Masters Degree Characteristics http://www.qaa.ac.uk/academicinfrastructure Race, P. (2001) A Briefing on Self, Peer & Group Assessment in LTSN Generic Centre Assessment Series No 9 LTSN York. Race P. (2006) The lecturer’s toolkit (3rd edition) London: Routledge.

36. Useful references 6 Race, P. and Pickford, R. (2007) Making Teaching work: Teaching smarter in post-compulsory education, London, Sage Rust, C., Price, M. and O’Donovan, B. (2003). Improving students’ learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education. 28 (2), 147-164. Sambell, K., McDowell, L. and Montgomery, C. (2012) Assessment for Learning in Higher Education Abingdon, Routledge Seymour, D. (2005) Learning Outcomes and Assessment: developing assessment criteria for Masters-level dissertations. Brookes eJournal of Learning and Teaching 1(2). Stefani, L and Carroll, J. (2001) A Briefing on Plagiarism http://www.ltsn.ac.uk/application.asp?app=resources.asp&process=full_record&section=generic&id=10 Yorke, M. (1999) Leaving Early: Undergraduate Non-completion in Higher Education, London: Routledge. Wharton, S. (2003) Defining appropriate criteria for the assessment of master's level TESOL Assignments, Assessment & Evaluation in Higher Education, 28(6), pp.649-664.

37. These and other slides will be available on my website at www.sally-brown.net