Evaluating Surgical Skills and OR Performance

1. Evaluating Surgical Skills and OR Performance Gary Dunnington MD Association of Program Directors in Surgery San Antonio April 21, 2010

2. Evaluating Surgical Skills and OR Performance • Some general principles of evaluation • Verification of Proficiency (VOP) for the Skills Lab • Evaluating performance in the OR • Milestones for the continuum of surgical training

3. Educational Framework Evaluate Program/Curriculum Instructional Method Instructional Materials Goals Objectives Assess Needs Educated Learner Teach Performance Evaluation Methods

4. Problems in Resident Evaluation • Inadequate sampling of performance (number of ratings, raters and skills sampled) • Inaccuracies due to overreliance on memory • Hidden performance deficits • Lack of meaningful benchmarks • Hesitancy to act • Systematic rater error Forecasting Residents’ Performance – Partly Cloudy Williams, Dunnington, Klamen, Acad Med, May, 2005

7. Deliberate Practice • Well defined task • Appropriate difficulty level for the particular individual • Informative feedback • Opportunities for repetition and correction of errors Ericsson, Psychological Review, 1993

10. From Practice Arena to Performance Arena

11. Rationale for Practice Arena before Performance Arena • Avoids exposing patients to the sharpest slope of the surgical skills learning curve • Enhances the quality of OR teaching • Potential to significantly decrease the number of operative cases needed for competency • Hour for hour, it may be more efficient than OR training for early trainees (Darzi)

13. NO EVALUATION During Teaching Modules or Practice Sessions

14. Evaluation of Skills Lab Learning OSATS (Objective Structured Assessment of Technical Skills) Versus Verification of Proficiency Modules

16. Asepsis, instruments Knot tying Suturing Tissue handling, wound management Advance tissue handling, flaps, grafts Catheterization Airway management Chest tubes Central lines Surgical biopsy Vascular anastomosis Laparotomy Bone fixation, casting Inguinal anatomy Upper endoscopy Colonoscopy Basic laparoscopic skills Advanced laparoscopic skills Hand sewn GI anastomosis Stapled GI anastomosis Phase I Curriculum Modules

17. Knot tying Basic suturing Chest tube placement Emergency surgical airway Basic laparoscopic skills Laparoscopic cholecystectomy Central venous access Bowel anastomosis Arterial anastomosis EGD Colonoscopy SIU Verification of Proficiency Modules

18. Verification of Proficiency Modules • Review of video of expert performance • Guided practice until performance within time standards • Video of performance • Blinded review by expert faculty with “pass” or “needs more practice” • OR performance ONLY after verification (proficiency = OR ready)

19. Implementation Strategy for Phase I at SIU Surgical Skills Boot Camp for all PGY I residents in Ortho, Plastics, Urology, ENT and General Surgery Twelve weeks beginning third week of July Three sessions weekly, each 90 minutes Nine of eleven verification of proficiency modules completed by last session

20. SIU PGY I Residents VOP Performance 2008-2009 (n=23) 2008 2009 Overall (n=12) (n=11) Failed at 83% 45% 65% least 1 VOP Failed at 75%* 18%* 48% least 2 VOPs *p < 0.05

21. Pass Rates for VOPs (%) VOP 2008 2009 Central line 75 73 Chest tube 100 100 Emergency airway 83 73 Simple interrupted suture 83 91 Deep one handed knot 58 91 Surface two handed knot 75 100

22. Remediation for Failed Lap Chole VOP • Assignment of faculty mentor (advisor) • Review of video with rater annotations • Practice with mentor feedback • Retest

23. Evaluation of OR Performance • The standard has been end of rotation, global rating of OR performance • Relies on remote memory • Details may be lost or overrepresented

24. SIU Operative Performance Rating System (OPRS)

25. Operating Room Performance Ratings (Sentinel Case Mapping) • Rating forms for selected, representative procedures • Evaluation of case specific items as well as global items • Records case difficulty and faculty supervision level • System provides for early feedback on resident performance thru New Innovations Larson, Williams, Dunnington, et al, Surgery, Oct., 2005

26. Excisional biopsy Open inguinal hernia Lap chole Dialysis graft Colon resection Mastectomy/partial Thyroidectomy Parathyroidectomy Lap inguinal hernia Lap colectomy Ventral hernia Lap ventral hernia Lap appendectomy Documentation of Sentinel Case Proficiency Proficiency achieved with four “competent” evaluations from at least two raters

27. Open Inguinal Hernia Instrument Search for indirect hernia 1 2 3 4 5 Moderate efficiency in search Poor technique in search for hernia Careful, meticulous search with high ligation Time and motion 1 2 3 4 5 Many unnecessary moves Clear economy of motion and maximum efficiency Efficient time/motion but some unneeded moves

28. Average Resident OPRS Ratings by PG Year for Lap Chole

29. The Current Debate in Rating of Operative Performance • Global rating instruments with specificity perhaps only to laparoscopic skills versus • Case specific rating instruments • As for OSATS research, “global” appears to fare as well as “specific” for reliability • Case specific instruments provide specific feedback to residents and faculty development

30. Data Capture System for Performance Evaluation • SIU alpha pilot site for Meti-Vision • Cameras for web-based data collection in skills laboratories, OR and simulation suites • Performance video sent to faculty raters with attached rating instrument • Performance evaluation stored in trainee evaluation portfolio

32. OR Integrated Video/Audio System • OR light camera • Laparoscopic/head mounted camera • Three team view cameras • Sound mixer (four wireless mics) • Physiologic data from anesthesia Ability to select four of eight connections simultaneously to the METI box

33. Developing a National Operative Performance Evaluation System • ABS will require operative performance evaluations for certification in the near future • SCORE selected SIU to develop system and validate nationally • Standard setting with “gold standard surgeons” evaluating video performances (7 per procedure) followed by national piloting for norms • Procedures: Lap chole, open inguinal hernia repair, lap ventral hernia, thyroidectomy

34. Effects of Guidance and Direction ABS Project • We believe there are two levels of OR proficiency to consider • Level 1: resident ability to perform psychomotor elements of procedure under faculty direction (early years of training) • Level 2: resident ability to plan and conduct procedure (making decisions about how and when to proceed, directing OR team including attending)

35. Conclusions From the First Gold Standard Session (ABS Project) • Judges arrived at agreement on the quality of virtually all performance dimensions in each of the four cases • There is reasonably good consistency between independent individual judge ratings and gold standard ratings • Where consistency was low on one case it was acceptable on the other similar case suggesting that ambiguities are in performances not in the rating forms

36. SIU Milestones for Measuring Performance in Operative Skills • PGY I, II: Verification of Proficiency, 11 modules • PGY III:FLS certification • PGY III, IV, V: Operative Performance Rating System (OPRS), 8 procedures