New challenges in assessment of clinical competence

1. Neil SpikeDirector of Training & AssessmentSchool of Primary Health CareMonash University, Australia New challenges in assessment of clinical competence

2. Answers? We’re not actually going to tell you how to play the game; that’s something you’ll have to figure out for yourself. After all the object of the game is to discover The Meaning of Life, and life does not come with an instruction book. If it did, you’d be wasting all your days in the troubleshooting section, hoping to figure out where you went wrong. That would be no fun. Manual for Monty Python’s Meaning of Life CD-Rom game.

3. Answers? If you have the same ideas as everybody else but have them one week earlier than everyone else then you will be hailed as a visionary. But if you have them five years earlier, you will be called a lunatic. Barry Jones

4. Questions? Confidence never comes from having all the answers; it comes from being open to all the questions. – Earl Gary Stevens

5. Overview • Other parameters of utility • Changes in students • Changes in academic roles • Performance assessment • General professional competencies

6. Overview • Other parameters of utility • Changes in students • Changes in academic roles • Performance assessment • General professional competencies

7. Utility function • U = Utility • R = Reliability • V = Validity • E = Educational impact • A = Acceptability • C = Cost • W = Weight U = wrR x wvV x weE x waA x wcC Adapted from Cees van der Vleuten

8. Utility function • U = Utility • R = Reliability • V = Validity • E = Educational impact • A = Acceptability • C = Cost • W = Weight U = wrR x wvV x weE x waAxwcC

9. Utility function • Introduce a new factor: • Feasibility

13. Overview • Other parameters of utility • Changes in students • Changes in academic roles • Performance assessment • General professional competencies

14. Universiteit Maastricht Student-centred assessment -self-assessment self-evaluation skills assessing the unmeasurable - peer-assessment peer-evaluation skills peer pressure? assessing the unmeasurable - co-assessment self, peer and co-evaluation skills peer pressure? joint responsibility

15. Student changes • Increasing diversity • Ethnicity • Socio-economic status • Previous school/educational experience • Equity categories • Basis of admission Adapted from Frank Tesoriero

16. Student changes • Student diversity means that previous assumptions on student characteristics may no longer hold true • Common values • Preferred learning styles • Levels of language proficiency

17. Student centredness • Early needs assessment • Individual learning plans & portfolios • Changes to staffing ratios • Professional development for students • Increased time for increased assessments • Collating numerous assessments • Translating the standard into a cut score

19. Overview • Other parameters of utility • Changes in students • Changes in academic roles • Performance assessment • General professional competencies

20. Universiteit Maastricht Suggestions Assessment is not mainly a measurement problem but more an educational design problem. Masterplan of the assessment system - built-in quality control procedures and bodies - sampling in stead of (over) structuring - variation of methods, compensation, regulations - saturation of information approach - training of students, teachers, examiners, board members

21. Academic changes • In addition to traditional roles: • Research • Clinical service • Teaching • Consultancy • Management of casualised teaching & assessment teams

22. Academic changes • Need to maintain program integrity when teachers are casual and teach outside of a full program context. • Need to adequately select, induct, support and manage of casual staff both individually and collectively. • Need to engage casual staff who are often at margins of academic life, in the teaching and learning debates which have significant impact on them, their teaching and their students. Adapted from Frank Tesoriero

23. Overview • Other parameters of utility • Changes in students • Changes in academic roles • Performance assessment • General professional competencies

24. Broaden the measurement time frame Universiteit Maastricht Specific test occasions - specific preparation of students - inflexibility - unwanted influence on learning - certain aspects unmeasurable Longitudinal testing - no specific preparation - congruent with flexible curricula - wanted influence on learning - more performance assessment

25. Performance Assessment • Indirect methods • Chart audit • Practice audits • Prescription data……… • Direct methods • Clinical (supervisor) ratings • Masked standardized patients • Video assessment • Patient reports • Peer reports • Clinical work samples………

26. Overview • Other parameters of utility • Changes in students • Changes in academic roles • Performance assessment • General professional competencies

27. dealing with tasks planning and time management information gathering dealing with others giving feedback and criticising communicating co-operation dealing with oneself self-responsibility self-evaluation self-reflection Universiteit Maastricht General professional competencies

28. The aim of education is the knowledge not of facts but of values. William Ralph Inge

29. Professional competencies Quality assurance focus • Traditionally includes: • The curriculum • The teaching and learning process • The assessment of competence • Should also include: • Monitoring • Evaluation • Staff development • “Fitness for purpose”

30. Where do we start? ‘Would you tell me please, which way I ought to go from here?’ ‘That depends a good deal on where you want to get to,’ said the Cat. ‘I don’t much care where’ said Alice. ‘Then it doesn’t matter which way you go,’ said the Cat. Alice’s Adventures in Wonderland - Lewis Carroll