Does resident recruitment ranking predict subsequent performance?

1. Does resident recruitment ranking predict subsequent performance? Jonathan Fryer, Noreen Corcoran, Brian George, Ed Wang, Deb DaRosa Northwestern University Feinberg School of Medicine, Chicago Illinois

2. Resident Selection Process

3. Questions? • 1) How effective is the ranking process in selecting residents who will perform well? • 2) How effective is the ranking process in predicting subsequent resident performances? • 3) How effective are “adjustments” made to the preliminary rank list in improving the selection of residents who will perform well?

4. Methods • General surgery categorical resident recruitment between 2002-2011 inclusive (n=46). • 4 categorical residents 2002-2004. • 5 categorical residents 2005-2011. • Residents who dropped out (n=1) or who were recruited after the PGY1 year (n=2) were excluded from the analyses.

5. Methods • We compared how successful candidates were ranked during recruitment with their subsequent performance in our program.

6. Recruitment Ranking Parameters • 1) USMLE˄Scores alone • 2) UnadjustedRanking Score (URS˄): based on sum of 3 assessment scores • Academic Profile (Coordinators) • Medical school rank, USMLE Step 1, Class Rank, Honors in Surgery • Program Director Review (PDs) • Research experience, extracurricular/community involvement, LORs, Personal Statement, Dean’s Letter • Faculty Interview score (Faculty) • Averaged for 2 independent faculty interview scores. ˄ Higher score is better

7. Recruitment Ranking Parameters • 3) Final Adjusted Ranking (FAR˅) • Modification of the preliminary rank list generated by the URS • Based on additional insights about specific candidates provided by the resident selection committee and/or leadership • Endorsements from trusted colleagues • Negative interactions with staff • Concerns raised by residents, coordinators • Other? ˅ Lower is better

8. Resident Selection ProcessNUFSM *USMLE included

9. Resident Performance Measures • 1) ABSITE˄ percentile alone • 2) Resident Evaluation Grade (REG˄) • Semiannual evaluation scores (Letter grade: A-F) • Group discussion and grade assignment based on: • Clinical Evaluations (360°): faculty, peers, med students, nurses, patients, etc. • Compliance: evaluations, case log, duty hour log, conference attendance, etc. • ABSITE, Mock Oral, PAME scores. ˄ higher is better

10. Resident Performance Measures • 3) Independent Faculty Rating/Ranking (IFRR˅) • Confidential survey with faculty independently rating all residents using a 7-point Likert scale and ranking resident within theirPGY1 recruitment cohorts. • * not part of standard resident evaluation at NU ˅ lower is better

11. Methods • Full IRB approval was obtained. • All resident ranking and performance data was de-identified after collection and aggregated to protect resident confidentiality.

12. Methods • Semiannual Resident Evaluation Grades (A-F) were converted to numerical values (5-0, respectively) and averaged for analyses. • Data from Individual Faculty Rating/Ranking surveys were averaged for individual residents.

13. Methods • Statistical Analyses performed using SAS 9.2 software (Cary, NC). • Associations between ranking and performance parameters were analyzed using Spearman’s correlation coefficient. • Comparison of ranking parameters between poor and satisfactory performance used student t-test. • Differences in performance based on ranking range were compared using F-test.

14. ResultsOverall resident performance * Occurring at any time during residency training

15. Recruitment Ranking vs Performance ˄ higher is better ˅ lower is better * Spearman correlation coefficient

16. Predicting Poor Resident PerformancesPredictors of Poor Resident Performances * Occurring at any time during the residency

17. Predicting Poor Resident PerformancesPredictors of Poor Resident Performances * Occurring at any time during the residency

18. Rank range vs. performance(URS)

19. Rank range vs. performance(FAR)

20. Rank range vs. performance(FAR)

21. Study Limitations • Single center study • URS confounded by USMLE score • REG confounded by ABSITE • No formal faculty orientation for IFRR survey

22. Summary • USMLE scores were predictive of subsequent ABSITE performance only. • Unadjusted Ranking Scores (URS) were predictive of subsequent performance based on resident evaluation grades (REGs), while Final Adjusted Rankings (FAR) were not.

23. Conclusions • Our resident selection process has generally been successful in providing us with residents who perform well. • Our unadjusted ranking scoreappears to be a better predictor of subsequent resident performance than our final adjusted ranking… ….therefore caution should be exercised when considering adjustmentsto the preliminary rank list, as they may not engender selection of better performing residents. • Effectively defining a reliable rank list “cutoff”, beyond which performance will predictably decrease, may not be possible in our system.

27. ResultsOverall resident performance • Drop outs: 1 (after PGY1 despite excellent performance) • ABS exams first try pass rate (n=13): • 100% • Ever with REG <C 100% (i.e. probation): 1 • IFR > 4.0 (i.e. below average): 6 • Ever with ABSITE scores < 35%tile ever: 12

28. Range range vs. performance

29. Predicting Poor Resident PerformancesPredictors of Poor Resident Performance

30. Results Absolute Ranking • Absolute ranking correlation with resident performance: • Absite • Semiannual evaluation grade • Faculty survey rating

31. Recruitment Ranking • Absolute ranking (AR): Ranking among entire candidate group (n= 60-80). • Relative ranking (RR): Ranking among cohort of successful PGY1 applicants (n=4 or 5).

32. Results (ABSTRACT)Relative Ranking • Within resident cohorts FAR did not correlate significantly with subsequent: • ABSITE scores (r=0.22; p=0.1760) • Semi-annual evaluation scores (r=0.20; p=0.1987) • Faculty survey cohort rankings (r=0.23;0.1175) • Conversely, USMLE scores exhibited a significantly positive correlation with subsequent: • ABSITE scores (r=0.46; p=0.0022), • Semi-annual evaluation scores (r=0.41; p=0.0163) • Faculty cohort rankings (r=0.35; p=0.163)

33. Introduction • Resident recruitment involves a formal evaluation of candidates where a variety of objective and subjective criteria are used to rank candidates from best to worst. • Preliminary rank lists are often subsequently “adjusted” based on additional insights about the candidates.

34. Resident Selection ProcessNUFSM