Poor Correlation Between Faculty and Resident Evaluations and Exam Scores in Surgery Clerkship

1. Faculty and Resident Evaluations of Medical Students on a Surgery Clerkship Correlate Poorly with Standardized Exam Scores Seth D. Goldstein, MD1, Brenessa Lindeman, MD1, Jorie Colbert-Getz, PhD3, Trisha Arbella1, Robert Dudas, MD2, Anne Lidor, MD, MPH1, Bethany Sacks, MD, MEd1 Johns Hopkins Hospital Departments of Surgery1 and Pediatrics2 Johns Hopkins School of Medicine Office of Medical Education Services3

2. No gold standard exists for medical student clerkship evaluation Survey Clerkship Assessment

3. Mixed conclusions in prior studies of objective-subjective student clinical knowledge correlation Objective and Subjective Knowledge Correlation Survey

4. Study Aim • Investigate convergent validity between subjective ratings of surgery student clinical knowledge and scores on the NBME subject examination • Inter-rater reliability of faculty and resident evaluations of global clinical knowledge

5. Study Design Retrospective, cross-sectional study from July 2009 – June 2011 Subjective Student Evaluation Scale Survey

6. A multi-faceted approach was employed to correlate subjective and objective student evaluations (ρ)

7. Inter-Rater Reliability Survey Intraclass Correlation Coefficient 0.76 0.81

8. Convergent Validity Survey ρ2: Attendings <0.01 Residents 0.01

9. Quartile Comparison Medical Student Ratings by NBME Percentile Group Survey p=0.007 Bottom 25th Middle 50th Top 25th

10. Conclusions 1Pulito AR, Donnelly MB, Plymale M, et al. What do faculty observe of medical students' clinical performance? Teach Learn Med 2006;18:99-104.

11. Conclusions and Future Directions

12. Thank You