Measuring Outcomes for Residency Graduates

1. Measuring Outcomes for Residency Graduates Steven L. Frick, MD Chairman, Dept. of Orthopaedic Surgery Director of Medical Education, Nemours Children’s Hospital Professor and Assistant Dean University of Central Florida College of Medicine Orlando, Florida

2. No financial disclosures. AAOS Program Committee POSNA Curriculum Committee POSNA Residents Review POSNA Treasurer, Board of Directors JRGOS Board of Directors ABOS QWTF ACGME Milestones Project Workgroup

5. No competency specifically addressing surgical skills

6. Macy Foundation Report 2011The Content and Format of Graduate Medical Education Recommendation III-A: The length of GME should be determined by an individual’s readiness for independent practice- demonstrated by fulfillment of nationally endorsed, specialty-specific standards- rather than tied to a GME program of fixed duration.

7. “nationally endorsed, specialty-specific standards” Do we have any of those?

8. Role for CORD • Optimistic versus Pessimistic • “an opportunity in every difficulty” versus “ a difficulty in every opportunity”

9. Evidence Based Medicine Outcome A final product or end result Integrating individual clinical expertise with the best external clinical evidence

11. A. Flexner - 1910 Medicine can be learned but not taught Active participation required Need dedicated educators and students

12. “get comfortable with uncertainty”

13. Role of professional education • Provide practitioners the intellectual tools to assess information critically, stay abreast of changing knowledge, adapt to continuous change, and reflect on the larger role and responsibilities of the profession in society. • From Time to Heal by Kenneth Ludmerer

14. “Is there a core body of knowledge and skills that the finishing resident should possess prior to starting practice or fellowship?”- Richard Gross, MD Need curriculum and competency assessment

15. Residency fundamentally =Master-Apprentice

16. William Halsted:Residency Training system Introduced in 1889 at Johns Hopkins based on: • a fixed period of time for training, • structured educational content, • actual experience with patients, • escalating responsibility for patient care during training, and a period of supervised practice after formal training. • Remains the cornerstone of surgical training in North America more than a century later

17. Competency Based Education • Defined by the outcome of the educational process, not the content • Develop weighted curriculum to teach and assess (Farmer, Gross, Wadey) • Assessing competence focuses on what the learner is able to do

18. How do you assess competency? "the state or quality of being capable or competent; skill; ability."

20. Miller’s model of competence Professional authenticity Performance or “hands on” Does Live Demo; Multimedia Shows how Knows how Read, Listen Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

21. OCAP

22. Dreyfus Model of Skill Acquisition

23. Mastery Learning Model -Bloom 1968

24. Becoming Expert “The 10,000 Hour Rule” • About 10 years- dedicated practice • Bill Joy- UNIX, Sun Microsystems; Mozart; The Beatles; Bobby Fischer;Bill Gates

25. Model of complete clinical care Engage Find It Empathize Educate Fix It Enlist Opening Closing

26. Culturally Competent Care The ability to understand and work with patients whose beliefs, values, and histories are significantly different from our own.

27. Cultural Competence in Health Care CCC Education Team Harmony & Quality of Work Life Medical/Surgical Team Concerns • Quality of Outcomes • Patient-Physician Relationship • Malpractice Claims Error Prevention Courtesy of A. White, III, MD

28. Defining / Teaching/ Modeling Professionalism most important Drs. Cruess body of work Hidden curriculum Social Contract Individual Awareness

29. Teach and Model Professionalism

30. Surgery- tripartite body of knowledgeFrank Wilson, MD • Preoperative - evaluation, indications, planning • Intraoperative - technical execution • Postoperative - immobilization, weight-bearing, PT • All 3 necessary for success

31. Ortho Surgical Education • Interns - pre and post operative care, framework of ortho fundamentals, closed management of fxs • PGY 2/3 - basic decision-making and psychomotor skills • PGY 4/5 - independent decision-making, subspecialty skills, integrate knowledge

32. Our Educational philosophy at CMC • Not training • Stimulus - Reaction vs Stimulus - Thought - Reaction • Create one-on-one master-apprentice situations • Graduated responsibility • ALWAYS supervised in highest risk activities (OR) • Have to spend enough time with them to know

33. I DON’T KNOW How do you assess competency?

34. Charlotte Competency Stages • Stage I - do not know anything cannot do anything, and know it • Stage II – know and can do a lot, but do not recognize what you do not know and cannot do DANGER • Stage III – know and can do a great deal, but realize there is much you do not know

35. “The beginning of a mountaineer’s career, when energy and enthusiasm outpace experience and judgment, is said to be the most dangerous part.” Photo by Guillaume Dargaud

36. Setting Standards • Job of Chair and RPD to set standards of excellence • Graduates of program should meet these standards in all core competencies

37. Assessing competency • Complete 5 years of orthopaedic surgery program under watchful eye of PD • 12 months PGY1 / internship, 48 months orthopaedic surgery • Evaluations and comment by faculty, peer evaluations, portfolio (presentations, courses, outcomes instruments), OITE, operative experience log • Consensus of PD and faculty

38. ABOSI believe this individual is capable of the competent independent practice of orthopaedic surgery. Steven L. Frick, MD Residency Program Director

39. An Expert- Knows • Knows WHAT to do • Knows HOW to do it • Knows WHEN to do it (and when not to) • Knows WHY to do it • Knows WHEN to ask for help • Knows WHAT we don’t know

40. Is it possible/desirable to define and measure competencyand thengraduate a resident before 5 years?

41. A Competency-Based Curriculumin Orthopaedic Surgery:From Idea to Implementation Markku Nousiainen, MS, MD, MEd, FRCS(C) Sunnybrook Health Sciences Centre University of Toronto

42. Current challenges in residency training • reduced training • opportunities for • residents } • reduction in work hours • reduced time spent in OR teaching surgical skills • reduction in wait times • improvement in patient safety

43. Competency-based education “Training process that results in proven competency in the acquisition & application of skills & knowledge to medical practice that is not simply dependent on the student’s length of training & clinical experiences”

46. “Much of what is counted does not count, and much of what counts cannot be counted.”

47. Problems • Toronto experience- 5 years, now all in for first time • Still no defined “curriculum” • More resource intensive than traditional pathway = costs more • Current environment of GME= very dependent on Medicare funding • Some predict reduction in Medicare GME funding under PPACA 2010

48. How much of residency education is experiential? Can we list / define everything you need to learn? Can we transfer knowledge gained from experience without making residents have the experience?

49. Duty hours 2003 • First ortho class with 80 hour work week- double failure rate on part I ABOS certification exam • Similar result 2011 exam takers • Why? • Does this exam measure competency? • Who do you want – 90%ile or 30%ile?

50. GME-Decade of Accountability • To patients by residents, faculty • Patient safety, Resident safety- RPD • To residents by faculty, institution • Societal demands for assurance of competency • Safe, Effective, Patient centered, Timely, Efficient, Equitable (IOM) • Increased requirements by oversight organizations – RPD time • Professional, ethical behavior demanded