The challenge of assessing students’ professional development and achievement

1. The challenge of assessing students’ professional development and achievement Mantz Yorke mantzyorke@mantzyorke.plus.com SCEPTrE, University of Surrey 17 March 2011

2. Text for the day: 1 Reasons for the frequent failure of summative assessment by teachers to live up to the expectations for it are most often given in terms of their preparation, or lack of it, for this part of their work. Many teachers have a narrow view of assessment and do not know how to respond to freedom to use evidence from students’ actions, projects and processes. Merely being required to follow given criteria or guidelines is not enough. Harlen (2005, p.249)

3. Text for the day: 2 ... there is a need to assess work-based learning through methods that are adequate, valid, and avoid undermining the nature of the learning, given that it will typically be issue-based, driven by the learner, and transdisciplinary. The aim of assessment is generally to assess learners’ progress as ‘map-makers’ or self-managing practitioners ... not to confirm their conformance as ‘mapreaders’. Lester and Costley (2010, p.566)

4. 1 Contexts

5. HE curricula and the development of professionalism Programme designed around workplace engagement Total Work placement as integrated curricular component Work placement as separate curricular component Some Proportion of student’s scheduled time Employment divorced from HE curricula Extra- curricular award; self- selected placement None None Partial Complete Level of curricular integration

6. 2 The challenges of assessing professional achievements

7. The nature of the challenge The development of professionalism is often situated and contextualised in ways that differ considerably from the development of learning in academic disciplines It is the ‘situatedness’ of professional behaviour that constitutes such a challenge for summative assessment

8. What should be assessed? Work-based methods of assessment target the practice-based elements of higher education study and collect information that is not so much concerned with performance in their normal practice[*] but is concerned with being able to think at a higher level about practice and understand the context well enough to be able to offer interventions into practices that have been researched, analysed, critiqued and evaluated. [...] [The assumption is] that assessments based on a practice that has subsequently been theorized by the practitioner are a better reflection of higher level learning than assessments of routine performance. Costley and Armsby (2007, p.29) * Save in some areas of professional practice, as the authors note.

9. 3 Theory and practice

10. The interplay between practice and theory Value system Practice Theory

11. Kolb’s learning cycle (modified) Concrete experience Active experimentation Reflective observation Abstract conceptualisation Note: Personal qualities and attributes are almost entirely obscured

12. A variation on Bruner’s ‘spiral curriculum’ Trajectory of learning CE AE RO AC

13. 4 Summative assessment

14. What kinds of problem are the focus?

15. ‘Generic’ capabilities; WIL/WBL/WRL Traditional assessment methods Standards broadly understood Academic disciplines What are appropriate assessment methods? How should the issue of standards be dealt with? Note: In some subjects (e.g. Nursing and Teacher Education) the relationship is closer than a grid like this suggests.

16. Educational objectives & their assessment Type of objective Problem Solution Assessment Instructional Specified SpecifiedPrescriptive Problem-solving Specified/Open Open ‘Expressive’ Open OpenResponsive Social Open Open NB: Some alleged problem-solving is essentially puzzle-solving, where there is a ‘right answer’. This fits best in Row 1.

17. Assessing professional behaviour ... • ... involves evidencing: • what knowledge, understanding, values and capabilities • the person can demonstrate when engaged in their • professional work • what s/he achieves in work situations • how s/he goes about her/his work • Highest or normal level of performance?

18. Assessment methods for professional behaviour • Before engaging • Behaviour in, and reflections on, simulations • Analyses of case studies • Assignments and examinations of various kinds • During • Direct observation of individuals (which may involve checklists) • Direct observation of group activities • After • Reflective analyses or commentaries of various kinds (e.g. logs or essays) • Reports on the professional activity (e.g. work placement) • Portfolios of experience (consider ethics and assessor time commitment) • An analytical digest of the portfolio content might be more practicable. • Presentations relating to the professional activity (various modes) • Interviews • Evaluation of artefacts produced during the professional activity

19. Who is doing the assessing? • Educational provider • Employer • Student • Do they have sufficient expertise? • Are there any sociological pressures? Not in all cases Conflict of interest Reluctance to fail

20. Some awkward questions How many academics can truly claim to have ‘recent and relevant’ workplace experience when they come to assess professional performance? Whereas academics possess disciplinary expertise, do they have the capability to assess the ‘soft skills’ valued by workplaces? Do academics see enough of an assessee’s performance to reach a valid judgement? Do workplace assessors have sufficient expertise as regards assessment? How should assessments from the workplace and academic aspects of programmes be combined?

21. 5(a) Some philosophical considerations: measurement v judgement

22. Assessing professional performance Assessing professional capability and performance requires post hocjudgement of standards reached against general expected learning outcomes It requires the (summative) assessor to have sufficient expertise to make the judgement Notes Totting up marks for different aspects of performance is inadequate and inappropriate Being successful in some aspects of professional performance is mandatory (e.g. safety; classroom management), whereas a more developmental approach may be adopted in respect of other aspects of performance

23. Some achievements are measurable… • … with tolerable reliability, and some not (unless cost • is not an issue) • Some intended learning outcomes (Knight’s ‘wicked • competences’) are inherently fuzzy (even if apparently • specified in tight terms), and resist measurement • Areas of difficulty include • the exercise of ‘soft skills’ • solving novel, or unbounded, problems • Judgement is difficult to fit into grade-based systems • (‘mapping’ or profiling is preferable to averaging)

24. Gold at the end of the rainbow? Introducing standardised and understood methods of assessing and grading these [generic] attributes, at the level of difficulty appropriate to the stage of the learning process, ensures that students better understand why they must learn particular things and also provides meaningful evidence to use as part of their future career activities. Australian Universities Quality Agency (2009, p.12) A strong echo here of the Australian ‘Mayer Report’ (1992)

25. Gold at the end of the rainbow? No. … the more general and generic standards descriptions become, the less useful they are for achieving the goals of ‘greater confidence’ and a ‘clearer picture’ of the meaning of final achievement grades. Krause (2009, p.2)

26. Utility Tight..................................................................Loose Specified learning outcomes The optimisation issue

27. 5(b) Some philosophical considerations: realism v relativism

28. Realist v Relativist approaches (‘Ideal types’) (In practice, it is never as cut and dried as this)

29. Assessment: two contrasting models • Measurement model • Validity • Reliability • Objectivity • Generalisability • Low ‘reactivity’ • Fairness • Efficiency • ‘Cheat-proofness’ • Utility • Intelligibility • Judgement model • Credibility • Dependability • Confirmability • Transferability • ? ‘Reactivity’ • Fairness • Efficiency • ‘Cheat-proofness’ • Utility • Intelligibility

30. 6 Shifting the paradigm?

31. Assessment: a ‘paradigm shift’? (after Kuhn, 1962) An academic community operates according to received beliefs. Normal assessment is based on such received beliefs. Normal assessment attempts to force student achievements into the currently-used conceptual boxes. Normal assessment often suppresses novel approaches because they tend to subvert its basic tenets. A shift (a ‘scientific revolution’) in professional commitments to shared assumptions takes place when an anomaly subverts the existing tradition of assessment practice. New assumptions (paradigms) require the reconstruction of prior assumptions. This is difficult and time consuming. It is also strongly resisted by the established community.

32. Assessment: how the paradigm might shift? (Again, after Kuhn, 1962) Initially, a new potential paradigm may have few supporters. Proponents should work to improve it; explore it; and show how it can be effective in practice. The potential paradigm has to convince, and to gather support. Activities based on the paradigm multiply. Eventually, the new paradigm supplants the old.

33. Assessment: why the paradigm might not shift Less stable Any or all of the following might happen: ‘It ain’t that broke; let’s stick with it.’ ‘It’s too difficult to change.’ ‘It’s impractical.’ The messenger is discredited or, metaphorically, shot. Energy barrier Now More stable Possible future Time

34. 7 Some challenges

35. Some challenges for assessment practice • HEI’s involvement: standards, criteria and grading methodology • The sub-heading understates the inherent complexity! • Dealing (inter alia) with achievement within, and by, groups • Staff development in respect of assessment • Employer involvement in assessment of professional achievement • Training • Commitment to full engagement in the assessment process • Potential role conflict of mentor and assessor • Quality assurance re assessment of professional achievement These challenges have no easy answers