1 / 21

Resident Assessment Do OBJECTIVE MEASUREMENTS correlate with competency?

Resident Assessment Do OBJECTIVE MEASUREMENTS correlate with competency?. Steven Kane, MD David Robinson, MD Atlanta Medical Center. Resident Assessment Tools. Core Competencies Prior USMLEs Interviews Faculty Evaluations Intra-operative observation Clinical observation

rendor
Download Presentation

Resident Assessment Do OBJECTIVE MEASUREMENTS correlate with competency?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Resident AssessmentDo OBJECTIVE MEASUREMENTS correlate with competency? Steven Kane, MD David Robinson, MD Atlanta Medical Center

  2. Resident Assessment Tools • Core Competencies • Prior USMLEs • Interviews • Faculty Evaluations • Intra-operative observation • Clinical observation • Surgical Skills lab • Peer to Peer • Pre medical experience • Ancillary staff evaluations • OITE • Milestones

  3. Objective vs. Subjective 9.63

  4. Does a superior knowledge base translate to a superior Orthopedic Surgeon?

  5. Is anything more accurate than “I know a good resident when I see it?” • “I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description and perhaps I could never succeed in intelligibly doing so. But I know it when I see it……”1964 Potter Stewart

  6. Relation of Residency Selection Factors to Subsequent Orthopaedic In-training Examination Performance • Carmichael, Kelly D. MD; Westmoreland, James B. MD; Thomas, John A. MD; Patterson, Rita M. PhD • May 2005, Southern Medical Journal • The only statistically significant correlations to OITE scores were USMLE step I performance (220) and marital status (married)

  7. Do Scores of the USMLE Step 1 and OITE Correlate with the ABOS Part I Certifying Examination? • Paul J. Dougherty MD; Norman Walter MD; Peter Schilling MD; SoheilNajibi MD, PhD; and Harry Herkowitz MD • March 2010, Clinical Orthopaedics and Related Research • Those who averaged in the 27th percentile or lower on the OITE had a 57% chance of failing the ABOS Part I examination • Correlation coefficient of 0.53 between the USMLE Step 1 and ABOS Part I • Average correlation coefficient of 0.50 for postgraduate years (PGY) 2 through 5 OITE scores and ABOS Part I

  8. Passing the Boards: Can USMLE and Orthopaedic In-Training Examination Scores Predict Passage of the ABOS Part-I Examination? • KLEIN, GREGG R. MD; AUSTIN, MATTHEW S. MD; RANDOLPH, SUSAN; SHARKEY, PETER F. MD; HILIBRAND, ALAN S. MD • May 2004, The Journal of Bone and Joint Surgery-American Volume • Correlation coefficient for the USMLE Step-I score and the ABOS Part-I percentile score was 0.38 (p = 0.002) • Correlation coefficients were also calculated for the OITE percentile scores for years 3 (r = 0.52, p < 0.0001), 4 (r = 0.49, p < 0.0001), and 5 (r = 0.69, p < 0.0001). • There was a high risk of failure (63%; five of eight failed) on the ABOS Part-I examination when a resident scored below the twenty-ninth percentile for PGY-3 and below the twentieth percentile for PGY-5

  9. Does the objective measurement of knowledge (USMLE, OITE, ABOS), correlate to a Program Director or Chair’s opinion of a resident’s competency scores? • 5 programs evaluated176 current and former residents evaluated • Chairs or PD’s asked to rate overall competence and surgical competence along with the 6 core competencies while “modestly blinded” to USMLE, OITE, and ABOS percentiles

  10. A statistical cross comparison of USMLE 1, USMLE 2, OITE 1-5 percentiles, 6 Core Competencies, ABOS percentiles, Surgical Competency and Overall Competency upon graduation was compiled.

  11. Spearman p or R values:0-0.09 No Correlation0.1-0.3 Small Correlation0.3-0.5 Medium Correlation0.5-1.0 Strong Correlation

  12. OITE to ABOS

  13. A high medical knowledge core competency does correlate with a high ABOS score (Strong Correlation)

  14. ABOS was more correlative with surgical and overall competency than OITE (medium correlation)

  15. OITE YIT- 4 Appears to be a throw away year

  16. The only objective score to correlate with surgical competency was ABOS I percentile (medium correlation)

  17. Overall Competency has no correlation with OITE YIT 1-5

  18. Conclusions • Our “mildly blinded” study of 5 programs, objective measurements have very little correlation with a program director or chair’s opinion of a resident’s overall competency, core competencies outside of medical knowledge or surgical competency upon graduation. • The one exception is the ABOS percentile and surgical competency which is mildly correlative.

  19. “When given a choice on who will operate on me or my children, with all other qualities being equal, I would choose the physician with the higher scores….” Steven Frick. June 2012

More Related