Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem

1. Shifting Paradigms: Competency-based Medical Education and the Quality of Care Problem Reynolds Meeting 2012

2. Disclosures • Employed by the American Board of Internal Medicine • I receive royalties from Mosby-Elsevier for a textbook on assessment • I am a member of the board of NBME and Medbiquitous • I serve on committees at the AAMC, ABMS, ACGME and NBME

3. The Quality of Care Problem

4. Teaching Vs. Non-Teaching Hospital Quality Shahian DM, Nordberg P, Meyer GS, et al. Contemporary performance of U.S. teaching and nonteaching hospitals. Acad Med. 2012; 87: online.

5. Care of the Vulnerable Elderly Study Lynn LA, et al. Acad Med. 2009.

6. Hospital Comparisons on Quality and Resource Use (Higher scores represent better performance) Non-teaching (N= 997) Teaching (N=186) Exemplary Teaching Hospitals Resource use Composite Score Quality Composite Score Source: L. Binder, CEO of Leapfrog Group, email communication, March 2010

7. “Every system is perfectly designed to achieve the results it gets.” Paul Batalden

8. Medical Education: Restraining Forces on Change

9. The Current “Miracle” of Medical Education Dwell Time

10. Medical Education Architecture1 1Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: time to confront our assumptions. Med Educ. 2011; 45(1):69-80.

11. Thomas Kuhn and “Normal Science” • “Normal science, the activity in which most scientists inevitably spend almost all of their time, is predicated on the assumption that the scientific community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost” Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.

12. Could the Same be True of UME and GME? • “Normal education, the activity in which most educators inevitably spend almost all of their time, is predicated on the assumption that the educational community knows what the world is like. Much of the success of the enterprise derives from the community’s willingness to defend that assumption, if necessary at considerable cost” Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press. Chicago. 1962. Pg. 5.

13. The “Big Assumption as Truth” • We operate on many assumptions that over time become “truth” without our testing or questioning the veracity of those assumptions • Test assumptions as assumptions • Immunity to change • Preservation of status quo through fear • More comfortable to stay with familiar even when status quo isn’t effective Kegan and Lahey. The Way We Talk Can Change the Way We Work; Immunity to Change.

14. Competency-based Medical Education:A Way Forward?

15. Effective Systems: Where Education Must Occur Nelson EC, et al. Quality by Design. 2007

16. Early Principles: CBME • World Health Organization (1978): • “The intended output of a competency-based programme is a health professional who can practise medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.” McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based Curriculum Development in Medical Education. World Health Organization, Switzerland, 1978.

17. Traditional versus CBME: Start with System Needs Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010 17

18. Competency-Based Medical Education …is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies1 1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical education: theory to practice. Med Teach. 2010; 32: 638–645

19. Outcomes-based Education: General Principles • Patient outcomes ≈Educational outcomes • Experience ≠ Expertise • Exposure and dwell time are not sufficient proxies for competence • You can do something a 100 times wrong and develop experience, but it’s still wrong! • Must engage in effective experiences • Critical role for work-based assessments

20. Need for System Approach:Assessment Perspective

21. Clinical Competency Committee • Periodic review – professional growth opportunities for all • Early warning systems • Structured Portfolio • ITE (formative only) • Monthly Evaluations • MiniCEX • Medical record audit/QI project • Clinical question log • Multisource feedback • Trainee contributions (personal portfolio) • Research project • Trainee • Review portfolio • Reflect on contents • Contribute to portfolio • Program Leaders • Review portfolio periodically and systematically • Develop early warning system • Encourage reflection and self-assessment Program Summative Assessment Process • Licensing and Certification • USMLE • American Boards of Medical Specialties Assessment During Training: Components Advisor

22. Systems-based prac Interpersonal skills and Communication Medical record audit and QI project Practice-based learning and improvement MSF: Directed per protocol Twice/year Structured Portfolio EBM/ Question Log Mini-CEX: 10/year Patient care Faculty Evaluations ITE: 1/year Medical knowledge Professionalism Multi-faceted Evaluation ■ Trainee-directed■ Direct observation

23. Model For Programmatic Assessment (With permission from CPM van der Vleuten) Intermediate Eval Intermediate Eval Training Activities Final Evaluation v v v v v v Assessment Activities Supporting Activities Committee Time = learning task = learning artifact = single assessment data-point = single certification data point for mastery tasks = learner reflection and planning = social interaction around reflection (supervision) = learning task being an assessment task also

24. Clinical Competency Committee • Periodic review – professional growth opportunities for all • Early warning systems • Structured Portfolio • ITE (formative only) • Monthly Evaluations • MiniCEX • Medical record audit/QI project • Clinical question log • Multisource feedback • Trainee contributions (personal portfolio) • Research project • Trainee • Review portfolio • Reflect on contents • Contribute to portfolio • Program Leaders • Review portfolio periodically and systematically • Develop early warning system • Encourage reflection and self-assessment Program Summative Assessment Process • Licensing and Certification • USMLE • American Boards of Medical Specialties Assessment During Training: Components Advisor

25. “Wisdom of the Crowd” • Williams, Teach. Learn. Med. (2005) • No evidence that individuals in groups dominate discussions. • No evidence of ganging up/piling on • Thomas (2011) – Group assessment improved inter-rater reliability and reduced range restriction in multiple domains in an internal medicine residency

26. Narratives and Judgments • Pangaro (1999) – matching students to a “synthetic” descriptive framework (RIME) reliable and valid across multiple clerkships • Regehr (2007) – Matching students to a standardized set of holistic, realistic vignettes improved discrimination of student performance • Regehr (2012) – Faculty created narrative “profiles” (16 in all) found to produce consistent rankings of excellent, competent and problematic performance.

27. *With permission CPM van der Vleuten

28. The Road Forward: Kelly Caverzagie