Innovation in a traditional world- In-training assessment in PGME Lord Cohen lecture, ASME 2010

1. Innovation in a traditional world-In-training assessment in PGMELord Cohen lecture, ASME 2010 Professor Charlotte Ringsted, MD, MHPE, PhD Director of Centre for Clinical Education University of Copenhagen and Capital Region DK, Rigshospitalet

2. Innovation in a traditional world • Innovation • Outcome-based education and in-training assessment in Postgraduate Medical Education (PGME) • Traditional world • Undergraduate Medical Education (UGME) • End-of-training assessments and exams • In-training assessment programme • Anaesthesiology in DK • Internal medicine and Child and Adolescence psychiatry • Conclusion • Lessons learned and future directions

3. Undergraduate Bachelor - 3 years Candidate - 3 years Postgraduate Internship 1½ year, Foundation 1 year Specialist education Introduction 1 year Specialist Residency 4-5 years Continuous prof. development Expert educations 1-2 years Various Danish Medical Education Basic Science Clinical Science Intern Intro-year Specialist residency Expert ed. CPD

4. PGME 1991 NBH rules, guidelines Goals and objectives Specialist societies Speciality courses Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation No exams PGME reform 2001 NBH rules, guidelines Goals and objectives CanMEDS framework Plus ’general courses’* Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation In-training assessment Innovation: OBE and ITA

5. EFPO project, 1992 Undergraduate education, Ontario, society’s needs, eight roles CanMEDs project, 2000 Postgraduate education, RCPSC, entire Canada, seven roles Challenge - The seven roles Medical expert Communicator Health advocate Collaborator Manager Scholar Professional Whole person

6. 3 persons per km2 125 persons per km2 No 3 Canada and Denmark – Red and white; Neighbours; Hans Island

7. Validity of CanMEDS roles • Survey among doctors in East DK • Responses from 3072 doctors • Roles important • Increasing confidence • Different specialitiesdifferent profiles Ringsted et al. Med Educ 2006 Importance Confindence*

8. Some challenge! What, how, when, who? Internship Intro-year Specialist Residency

9. What is competence? Competence is a habit of action • Competence = holistic overall capacity • ”The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served” Epstein and Hundert JAMA 2002 • Competency = specific capability • ”Focuses on performance of the goal-state of instruction; reflects expectations that are external to the programme; expressible in measurable behaviour; uses criterion standards for judging; informs learners and others about expectations” Albanese ME 2008

10. 1. Challenge: Defining outcomes • Describing competence • Too detailed – Be able to manage - Lists of procedural skills, diseases (+300 competencies) • Too general - The ‘other’ six roles - difficult to define • Disintegration of the concept ‘competence’ • Seven disciplines rather than an integrated, context-based concept of competence • Expectations at various levels of training? • Different specialties, different traditions?

11. Various levels - AnaesthesiologyDreyfus, Epstein & Hundert TASK Complex Atypical CONTEXT Complex systems Supervisor Patients: ASA groups 1-5 Surgery: Minor, medium, major EM: Stay alive till assistance arrives 5th year PERSON Master Expert TASK Complicated Typical TASK Simple Single CONTEXT Large teams Distant supervision 2-4 year CONTEXT Small teams Close supervision 1st yr PERSON Proficient Competent PERSON Adv. beginner Novice

12. Internal medicine – levels? About the difference between trainee levels: ”We pretty much do the same – patient encounters, ward rounds, ambulatory, etc. - The difference between levels is a matter of expectations - you are expected to take on a wider and deeper approach in managing the patients - and you are allowed to do more - make more decisions - simply because you are more experienced”... Ringsted et al. Med Teach 2006

13. Child & Adolescence psychiatry • ”We work in multi-professional teams – these are not stationary teams, but rather depending on the child and the situation.” • ”We learn from the team and gradually we become more respected members of the team. • Our legitimacy is being a doctor with the right to make certain decisions and prescribe medicine” What is wrong with this kid? Davis et al. 2008

14. Conclusion - Different specialities ’Brick laying model’ Technical task specialities ’Community of practice’ Social task specialities ’Rings in the water’ Cognitive task specialities

15. 2. Challenge – assessment • What, when, how, who? Before entry Start of rotation End of rotation After exit MMI, Oral, MCQ Mini-CEX DOPS ITER ABIM OSCE, Oral, MCQ During rotation

16. Why in-training assessment? • Postgraduate education is work-based • 50% of the physician work-force are trainees • Quality of care relies on trainees’ competence during training • ”End-of training examination is like reading yesterday’s news” • In-training assessment, a tool for learning • Help clarify objectives according to broad aspects of competence (CanMEDS roles) • Stimulate deep learning • Support effective and efficient education

17. ? Challenge in postgraduate education Postgraduate education Does Work-based Does Can • Learn how to manage cases • Learn from managing cases • Reflect in and on practice Can School- based Knows Knows Undergraduate education

18. Mastery or Development ? Scoring Competency as capability related to specific tasks 9 8 7 6 5 4 3 2 1 • Competence as holistic • capacity related to any task • Aspects of the 7 roles Time 1. 2. 3. 4. 5. 6. No single method can measure it all – V.d .Vleuten 2010 assessment programmes are recommended

19. In-training-assessment programmes Anaesthesiology Internal Medicine

20. In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003 Clinical skills assessments (12) Observation in vivo / vitro • Cusum scoring • Logbook on experience • Learning portfolio 1 st year training Longitudinal assessment Assessment based on practice data and written reflective assignments/reports • Communication skills (1) • Management/collaboration (2) • Academic competence (3)

21. Assessment of written assignment • Reflection over a patient • Description of patient and operation • Theoretical and practical consideration regarding choice of anaesthesiological approach related to patient condition, wishes, surgery, and context • Describe potential problems and complications and discuss strategies to minimise these • Describe actual patient course and events • Reflection related to pre-operative considerations • Use references from literature in the reflection

22. Realistic orientation Recognition of uncertainty and unpredictability Communicative relationship: each patient is unique Objectivistic orientation No recognition of uncertainty and unpredictability Authoritative relationship: ’a case’: coronary, asthmatic, etc Why focus on theory and reflection? Klemola and Norros, ME 1997, 2001 • Anaesthesiology • Clinical physiology and pharmacology; Procedural skills; Monitoring of respiratory and cardiovascular parameters; Context – patient, surgery, team Two distinct patterns related to ‘experts’

23. Interpretative Combine monitor information with situational information and background knowledge Recognition of the versatility of information from several resources, oxygen SAT, End-tidal CO2, etc. Reactive Operate directly with the numbers Contradictory use of monitors, emphasising importance regarding patient safety without understanding the mediated character and versatility of information Habit of action Klemola and Norros, ME 1997, 2001

24. Knowledge and skills • Causal understanding of concepts, principles, and tool design affects retention and transfer of learning Woods et al. 2006, 2007, Schwartz 2004 • Self-regulatory processes in development of expertise Zimmerman 2006 • Forethought: Task analysis, strategic planning • Performance: Contextual adaptation of strategies • Post-task: Evaluation and reflection Bech et al. EJVS 2010

25. Knowledge and Anaesthsiology Klemola and Norros, ME 1997, 2001 • Forethought: physiological potentials • ”He can go uphill without getting out of breath, so probably he will tolerate anaesthesia well. Major problem might be oxygenation and ventilation.” • Adaptive strategy: physio-pharmacological experiment • ”You can’t tell how an elderly patient will react. You have to check his responses and give drugs accordingly.” • Evaluation and reflection-in-action • ”The patient has capacity to compensate for side-effects of anaesthesia through sympathetic activation, a kind of capacity that elderly patients do no necessarily possess. That is a safe thing to observe” Flexner ?

26. Routine expert vs. Adaptive expert Adaptive experts Performance Routine experts Experience ”Most professionals reach a stable, average level of performance and maintain this mediocre status for the rest of their careers.” A team and a coach Ericsson 2004, Guest et al.2001, Choudhry et al. 2005, Schwartz 2004

27. Assessment free area Focus on evaluation of quality of education “To emphasize the educational purpose of training, comprehensive formative evaluation is suggestedas alternative to specialist examinations.” Karle, Nystrup ME1995 Strong humanist tradition in education, qualitative R&D Deprived of quantitative educational researchand psychometrics Ministry of research, innovation, and technology 2007, OECD review 2005 3. Challenge: Validity and reliability?

28. Schuwirth & v.d. Vleuten ME 2006 A plea for new psychometric methods North America DK • Assessment rich area • Flooded by quantitative data and psychometricians • National exams • Heavy focus on reliability of tests and exams • Strong tradition of cognitive psychology and behaviourism • Hodges and Segouin, ME08

29. Trainees’ opinion of assessment (1-9) Ringsted et al. AAS 2003

30. Internal medicine Ringsted et al. Med Teach 2006; Davis et al 2005; Norgaard et al. Med Educ 2004 Intro-year trainees’ learning needs, case-mix, and quality assurance data • Structured approach to the tasks • Ward rounds • Complex patients – get an overview • Emergency care • Team-leader skills • Collaboration • Team-work and inter-personal skills • Ethics and professionalism • Difficult decisions

31. Usefulness of the assessment Median (1-9)

32. About the written assignments • “Extremely good learning experience - to do this review of a patient’s course ” • “It was hard work” (Trainee) • “This is really a valuable innovation in the education - these assignments” (Trainee) • “It was more easy than I thought - to review these assignments” (Supervisor) • “This is an advantage to the entire department - we all learn from these..” Kirsten Nørgaard, MHPE, 2004

33. Factors related to value of ITA Ringsted et al. ME 2004, Med Teach 2003, ASS 2003 • The link to practice • Help in structuring teaching, training and learning • Outcomes clear, monitoring progress, identify problems • Coupling of theory to practice • Used as licence to practice rather than end-of-training assessment • The effect on learning • Should include a challenge to the learner • ‘We all learn more’ • Assessors’ attitudes • Enthusiasm and rigour

34. Conclusion Lessons learned Future directions

35. Lessons learned • No 1 – Outcome-based education • ‘CanMEDs roles’ is a nice mental framework. Need for both competency-goals (outcome/ efficiency) and competence-goals (process/ innovation) • No 2 – In-training assessment • A valuable tool for learning. Meaningful programmes are tailored to clinical context and trainees’ level of professional development. • No 3 – The process • Useful to take a design-based research approach: Cycle of critical review of literature, design, enactment, evaluation, and large working groups

36. Future direction • Some questions • Can deliberate practice, reflection in and on practice, adaptive expertise be learned? Or is it an in-born trait in the minority of us? • If it can be learned, how can we facilitate the development and measure the progress of it? • How can we better align education, training, and learning with quality of practice and measure the effect? • How can we turn education into a resource by using the capacity of our trainees to develop practice as a whole?

37. In the honour of Lord Cohen Thank you for your attention ?? Time’s up