Northwestern University, Chicago, IL

1. A Clinical Curriculum Evaluation System That Promotes Ownership, Accountability, and Quality Improvements John Coyle, MD Mary Schuller, MSEd Ed Wang, PhD Debra DaRosa, PhD Northwestern University, Chicago, IL

2. Purpose • Few studies on effective strategies for improving multiple clinical rotations in a residency program • What is the effect of a feedback loop rotation evaluation system on a general surgery program?

3. Annual Rotation Evaluation Process LFMdesignsrotationobjectives Program Director Reviews improvement plan Lead Faculty Member(LFM) responsible for the rotation including creating learning objectives, complying with policies, obtaining feedback from other team faculty members about resident performance , meeting with resident to provide feedback at mid-end points of rotation LFM & Division Chiefs Reviews rotation performance summary Resident completes rotation evaluation Data aggregated & analyzed

4. Research questions Do rotations improve as a result of a feedback loop criterion-referenced evaluation system(FLES)? Which educational components are most effected, if any, by the FLES? What influences, if any, affect improvement or lack of improvement?

5. Data • 20 item rotation evaluation form 3 Q’s Operative experience 3 Q’s Supervision by learning site 4 Q’s Teaching 2 Q’s Continuity 4 Q’s Duty hour compliance 4 Q’s Miscellaneous • 5 years (2003/2004-2007/2008) • 45 rotations, 2076 forms, 201 residents • 90% response rate

6. Analysis 1: Percentage of scores on end of rotation evaluations that met acceptability criteria by year F=3.13;df=4; p=.01

7. Analysis 11 Which educational components were most effected?

8. Analysis II: Percent change in acceptable mean scores between 2003/2004 and 2007/2008 n/a= between 5% & -1%. change

9. Analysis II: Percent change in acceptable mean scores between 2003/2004 and 2007/2008 n/a= between 5% & -1%. change

10. Analysis II: Percent change in acceptable mean scores between 2003/2004 and 2007/2008 n/a= between 5% & -1%. change

11. Analysis II: Percent change in acceptable mean scores between 2003/2004 and 2007/2008 n/a= between 5% & -1%. change

12. Analysis II: Percent change in acceptable mean scores between 2003/2004 and 2007/2008

13. Rotations with 0-1 violations (non work hour) A Rotations with 2-3 violations (non work hour) B B Rotations with 4 or more violations and/or work hour violations or a seriously concerning remark. C C

14. A, B,& C rotations by year

15. Analysis I1I How does “consistency” effect rotation evaluations? "Consistency” = Stability of both LFM and resident: faculty ratio

16. p=.0018 n=40 n=5

17. Limitations • One institution • Not the same residents reviewing each year • Rotation faculty can change • Influence factor: Outside department rotations (40%)

18. Discussion • Highly specific feedback is critical to change • Submitting of a formal plan for improvement improves educational experience • Changes in rotation leadership & format disrupt improvement process • Most effected items related to teaching & subsequently supervision

19. Quotes “ Indeed I have been aware of these concerns for some time now. To that end, I made a decision to employ recurrent laryngeal nerve monitoring on a routine basis for my cases, so that I might be more confident in allowing the residents to do more dissection.” “ I am trialing the new Alexis wound retractor that will hopefully allow better exposure of the operative field and free up one of the resident’s hands so that they can do more operating.”

20. Questions ????