Workplace Based Assessments for Undergraduates: Training for Assessors

1. Workplace Based Assessments for Undergraduates: Training for Assessors Leighton Seal Responsible Examiner for WBA and Consultant Endocrinologist Dr Judith Ibison: Senior Lecturer Primary Care & MBBS Personal and Professional Development Theme lead, Dr Elizabeth Miles: Senior Lecturer Staff Development and CD PGCHE Dr Clare Shoults: T Year Lead and Consultant Acute Medical Unit

2. Workplace Based Assessments for Undergraduates: Training for Assessors Context of assessment for students WPBA How to supervise and assess WPBA Common pitfalls Video – try rating one Feedback

3. Outcomes 1 − The doctor as a scholar and a scientist • The graduate will be able to • Apply to medical practice biomedical scientific principles, method and knowledge relating to: anatomy, biochemistry, cell biology, genetics, immunology, microbiology, molecular biology, nutrition, pathology, pharmacology and physiology. • Apply psychological principles, method and knowledge to medical practice. • Apply social science principles, method and knowledge to medical practice. • Apply to medical practice the principles, method and knowledge of population health and the improvement of health and healthcare. • Apply scientific method and approaches to medical research.

4. Outcomes 2 − The doctor as a practitioner • The graduate will be able to • Carry out a consultation with a patient. • Diagnose and manage clinical presentations. • Communicate effectively with patients and colleagues in a medical context. • Provide immediate care in medical emergencies. • Prescribe drugs safely, effectively and economically. • Carry out practical procedures safely and effectively. • Use information effectively in a medical context.

5. Outcomes 3 − The doctor as a professional • The graduate will be able to • Behave according to ethical and legal principles. • Reflect, learn and teach others. • Learn and work effectively within a multiprofessional team. • Protect patients and improve care.

6. Why use workplace based assessments in medical students? • Importance of clinical skills • Deficiencies in clinical skills • Problems with traditional assessments • Effectiveness of feedback

7. Importance of clinical skills • Interview and PE are primary source of diagnostic data • Several studies document deficiencies • Data gathering errors are the most common source of diagnostic errors • Good communication skills improve health status of patients, increase efficiency of care and REDUCE likelihood of litigation

8. Feedback • Feedback is sought after and valued by medical students and trainees • Feedback is a strong motivator for behaviour change and has been shown to improve confidence and clinical performance • In meta-analyses of the impact of various educational strategies on medical education outcomes, feedback alone effective is effective in 71% of studies (Veloski et al. 2006)

9. Resources • http://www.elu.sgul.ac.uk/wba/

10. How to use the MBBS Workplace-based Assessment (WBA) resource • All users of the MBBS WBA materials on this website are strongly recommended to: • watch the “Introduction to WBA” video • watch the “Giving Effective Feedback” cluster of 3 videos (which set the scene for effective student learning through formative assessment)

11. How to use the MBBS Workplace-based Assessment (WBA) resource • Select additional resources to view by following the guidance after the statement (1-3) • 1. “I am unsure about WBA and what is expected of the assessor and the MBBS student.” • 2. “I already know about/have taken part in WBA at postgraduate level and want to know what standard to expect of the MBBS students.” • 3. “I already know about WBA and what level to expect, but want to check about formative assessment and giving feedback to MBBS students”.

12. How to use the MBBS Workplace-based Assessment (WBA) resource • Watch each of the three clusters of clips • WBA • WBA Examples • Feedback • (there are points and questions to consider beside each clip): • a. Case-based Discussion • b. Mini Clinical Examination • c. Directly Observed Procedural Skill

13. How to use the MBBS Workplace-based Assessment (WBA) resource • There are downloads attached to clips and under the Documents tab: • “Professional Behaviour & Clinical Practice Assessment Guide” • “Workplace-based Assessment” PowerPoint • “Changes to Assessment of St. George’s Students on Clinical Attachments” • WBA assessment forms

14. Effectiveness of feedback • Critical to learning • Significant influence on achievement – meta-analyses show efficacy of feedback alone in improvement • But traditionally students are rarely observed….

15. Promoting effective feedback • To facilitate the student’s own self-reflection. • Prevent the student having misconceptions about performance. • Ascertain whether there is disagreement or resistance to the observer’s comments.

16. https://intranet.sgul.ac.uk/teach/Clinical%20skills%20videos/cbd/cbd_bad.wmvhttps://intranet.sgul.ac.uk/teach/Clinical%20skills%20videos/cbd/cbd_bad.wmv

17. Problems that can occur with giving feedback include: • Non-specific feedback ‘clerk more patients’ • Giving personal or emotional response to performance • Fear of criticising the student. • Fear of emotional responses from student. • No time

18. https://intranet.sgul.ac.uk/teach/Clinical%20skills%20videos/cbd/cbd_good.wmvhttps://intranet.sgul.ac.uk/teach/Clinical%20skills%20videos/cbd/cbd_good.wmv

19. Promoting effective feedback • Optimise the value of feedback by: • Start with the learner’s self-assessment of what went well and what could be improved. • Reinforce important areas and points of specific feedback. • Review understanding and feeling in response to those areas and points. • Formulate a specific and realistic plan with clear, direct goals.

20. Workplace based assessments Miller’s pyramid WORK BASED ASSESSMENTS Objective Structured Clinical Examination Short Answer Question / Extended Matching Item Single Best Answer WPBA are one component of a comprehensive assessment strategy

21. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

22. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

23. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

24. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

25. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

26. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

27. Agreement between learner and assessor on focus of assessment Agreement of learning objectives and action plan for learner Suitable WPBA tool WPBA: ‘the learning moment’ Constructive feedback by assessor and rating Learner demonstrates skill by on or about real patient Assessor facilitates self appraisal by learner Self assessment by learner

28. What is a workplace based assessment? • WBA are observed assessments of clinical practice in the workplace, with feedback to the learner on the strengths and weaknesses of his or her performance. • The key is observed practice and developmental feedback.

29. Workplace based assessments • Vehicles for offering direct, timely and relevant feedback • High validity, • High acceptability, • High educational impact. • ? Cost –effectiveness • Poor reliability

30. Key components of a WPBA Designed to assess a specific skill/task Use rating scales – to provide information on specific behaviours and the quality and fitness of their performance (vs checklists) Opportunity for feedback to improve subsequent performance

31. Skills • History • Examination • Diagnostic Reasoning • Management – investigations and interventions • Ethics • Practical procedures • Acute or chronic contexts • General or specific • Personal behaviours relevant to clinical practice

33. WBA: Assessment challenges: reliability • Inter-observer variability • Intra-observer reliability • Case-specificity • Therefore we are using them formatively

34. Mini CEX • Should be based on • observed focused history taking, • observed examination • observed higher communication skills • Main domains • history taking, • physical examination skill, • clinical judgement and synthesis, • humanistic skills • Also overall rating • Different details in different schemes • number of items and rating scales

36. How many mini-CEX do you need? • In T year • 2 CEX per long clinical attachment • Surgery • GP • Medicine • 1 In AMU week

37. In P year 2 CEX per long clinical attachment 1 Short Attachments Exceptions O+G Psychiatry Medicine Surgery Senior Medicine Paediatrics AMU week Speciality Neurology 2 CEX (Including GCS) How many mini-CEX do you need?

38. How many mini-CEX do you need? • In F year • 1CEX per long firm • Surgery AHO • Medicine AHO • GP AHO

39. Direct Observation of Procedural Skills • DOPs allows students to move from procedures learnt in a skills lab to performance in real patients. • Performance of practical procedures involves technical skills but also communication, patient approach and attitudes to analgesia and patient comfort

40. DOPS 4P’s • Preparation • Performance • Practice Based Discussion • Progression

41. DOPS Assessment • Preparation for DOPS • Proficient in SIMS • Observation in work place of a competent practioner • Consent

42. DOPS Assessment • Performance • See Clinical Skills LOBS • Pre clinical and T year LOBS for clinical skills

43. DOPS Assessment • Practice Based Discussion • Indication • Safety • Interpretation of result

44. DOPS Assessment • Progression • Reflection on performance of the skill in a clinical setting • Learning needed • Further assessment

45. DOPS • DOPS compulsory but assessment formative. • Practical procedures must be supervised and assessed by F1 or above, or non-medic who performs procedure routinely in clinical practice e.g. nurse or e.c.g. technician Standard: Procedurally correct and safe

46. Supervising DOPS • Are you confident to do the procedure? If not refuse……. • Is the student ready to perform the procedure on a real patient? • Talk through whether the student knows how to do the procedure…..can they accurately describe the detail of the process……if not refuse!

47. End of Firm Assessment • Previously had problems Benchmarking results

48. Clinical Practice Grades by attachment

49. Clinical Practice: AHO Surgery by site No. of students

50. Clinical Practice: AHO medicine by site No. of students