Educational Outcomes Service Group: Overview of Year One

1. Educational Outcomes Service Group: Overview of Year One Lynne Tomasa, PhD May 15, 2003

2. Education of Program Directors and Faculty • Program directors were familiar with requirement and toolbox • Programs were mainly utilizing global assessment tools • Programs rarely utilized direct observation and evaluation • Evaluation tools were used mainly for summative feedback

3. Faculty Buy-In • Program directors did all or most of curriculum development, so group buy-in was minimal • Junior faculty members had limited experience in curriculum design/teaching • Programs with faculty at multiple sites had additional challenges

4. Curriculum Design • Program coordinators were helpful but had various amounts of expertise in this area • Many program directors were new and junior and had minimal experience in curriculum development

5. Six ACGME Domains • Overlapped with each other • Forced one to define it according to specialty • Patient care and medical knowledge were easiest to define and understand • Professionalism: “know it when you see it” but behaviors were not clearly documented

6. Six Domains • Interpersonal and communication skills made sense but often did not incorporate colleagues, staff, and other health professionals • Practice-based learning and improvement and systems-based practice were “fuzzy” and forced one to think outside the “box” • Systems-based practice was least intuitive • Health care professionals were now viewed as part of a team

7. Paradigm Shift • To Educational Outcomes • From Process and Structure • Compares learner’s performance with that of a peer group • Fails to provide clear understanding of what resident can or cannot do • Cannot determine if benchmarks or performance indicators have been met

8. Educational Outcomes • Clearly defines curricular objectives or benchmarks to describe competencies • Faculty must be more accountable and possess the skills/competencies themselves • Requires a structure to record, collect, and analyze data • More costly to implement

9. “Competency” Defined A complex set of behaviors built on the components of knowledge, skills, attitudes, and “competence” as personal ability. A complex but demonstrable integration of numerous related objectives, the latter being discrete measurable behaviors. Carraccio, Wolfsthal, Englander, Ferentz, Martin, Academic Medicine, May 2002

10. Curricular Design Stepwise Approach • Competency identification • Determination of competency components and performance levels • Competency evaluation • Overall assessment of the process Carraccio, et al. 2002

11. Step 1:Competency Identification • Consensus of individual experts • Group consensus • Task analysis (document all activities over a period of time) • Critical-incident survey (describe observed incidents that reflect good or bad practice) • Behavioral-event interview (star performers describe critical clinical situations and characteristics of a good doctor) • Practitioner surveys

12. Step 2: Competency Components • Identification of “tasks” that sequentially or in sum, make up the competency • Benchmarks or performance indicators • Must be measurable • In aggregate, determine achievement of the specific competency

13. Step 2: Performance Levels • Performance levels set threshold for demonstrating competence • Each benchmark must be clearly defined to determine whether competence has been achieved • Must determine the methods by which the competency might be attained

14. Step 3: Competency Evaluation • Criterion-referenced measures compare performance against a set standard or threshold are the preferred methods • Adult learners • Incorporates a variety of methods and strategies for adult learners

15. Program Directors’ Concerns • Where’s the evidence? • There is little • Studies have utilized small numbers • PD time commitment • Faculty buy-in and education

16. Responses to ACGME Requirements • I wish this goes away • I’ll wait until our specialty board tells us exactly what to do • I/We don’t have the time • I have a large faculty in a variety of sites • This is one more thing we have to do • How do we know that this shift will produce better, more skillful physicians?

17. Common Themes • Residents were not an active participant in identifying competencies • Residents have either none or minimal knowledge about the goals of the ACGME Outcomes Project • Program directors feeling “overwhelmed” at the enormity of task

18. Common Themes • Want clearer guidelines: no consensus on what outcomes are desired • On one hand, Program Directors want autonomy and on the other they want more direction • Question of what outcomes are desired

19. Examples of Desired Outcomes • Improved career satisfaction • Decreased medication errors • Decreased hospital stays • Better balance between personal and professional life • Longer life for patients • Better compliance with preventive health measures

20. Motivating Factors • RRC visit in the near future • Enthusiastic faculty with interest in competency-based curriculum

21. Non-Motivating Factors • Implementation of duty-hours takes time away from teaching and curriculum development • RRC visit occurred just prior to July 2002 • Lack of support from Director of Medical Education