“USMLE Step 2 Clinical Skills”

1. “USMLE Step 2 Clinical Skills” Richard E. Hawkins, MD National Board of Medical Examiners Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

2. Presentation Structure • Overview of USMLE Step 2 CS • Measures to ensure fairness and consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

3. Overview of USMLE Step 2 CS Purpose and Delivery Model Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

4. USMLE: Purpose • Three step examination system primarily designed to support the medical licensing process • Jointly sponsored by • Federation of State Medical Boards (FSMB) • National Board of Medical Examiners (NBME) • Step 2 CS: collaboration with ECFMG Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

5. USMLE: Purpose • Overall purpose • Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

6. USMLE: Purpose • Overall purpose • Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

7. USMLE: Purpose • Patient-centered Skills • History taking / Physical examination • Communication and Interpersonal Skills • Medical Record Documentation • Important for safe and effective patient care • History/PE  Diagnosis and Management • Communication  Patient Health Outcomes • Medical Record  Errors and Patient Safety Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

8. USMLE Step 2 CS • Enhancement to USMLE Step 2 • USMLE Step 2 – components: • Clinical knowledge (CK) • Clinical skills (CS) • Standard for Step 2 – appropriate for entry into postgraduate training Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

9. USMLE Step 2 CS: Logistics • Regional delivery model • Optimal combination of convenience, cost-efficiency and standardization • Five regional test centers across US • Projected examine volume – 30,000+ / year • Individual center capacity • 3 examinations / day (33 examinees); up to 7 days/week Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

10. Requirements of a High StakesPerformance-based Examination Ensuring Fairness and Consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

11. Threats to Validity and Reliability • Content and/or tasks not relevant or realistic • Individual test forms vary in content coverage • Scoring methods not appropriate for skills tested • Inconsistency in SP portrayal and scoring • Between cases, across sites, over time • Level of difficulty of cases / exams inconsistent • Standard setting approach appropriate Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

12. USMLE Step 2 CS: Content • Step 2 CS Blueprint • Defines content categories • In meeting blueprint specifications, each test form provides: • Adequate sampling of content domain • Comparable content between test forms Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

13. USMLE Step 2 CS: Content Blueprint Content Categories: • Common and important medical problems / patient presentations • Acuity • Age • Gender • Race / ethnicity Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

14. Case Content Cardiovascular Respiratory Gastrointestinal Musculoskeletal Constitutional Neurological Psychiatric Genitourinary Women’s health Unclassified / multi-system Case Acuity Acute Subacute / Chronic Test Form Patient age Age less than 18 Age 18 – 44 Age 45 – 64 Age 65 + Patient Gender Male Female Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

15. Step 2 CS: Structure & Tasks • 12 patient encounters • 15 min. for encounter / 10 min. for patient note • Each encounter: • Elicit pertinent history, Perform appropriate physical examination, Communicate effectively • Document: • Findings from the history and physical • Diagnostic impression / Further work-up Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

16. Step 2 CS: Score Components • Integrated Clinical Encounter (ICE) • Data gathering: history and physical exam • Patient note • Communication / Interpersonal Skills (CIS) • Gathering information; sharing information; manner & rapport • Spoken English Proficiency (SEP) • Listener effort, examinee pronunciation / word choice Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

17. Scoring Approach: ICE • Data gathering (Hx / PE) • Dichotomous checklists completed by SPs • + • Patient note • Physician raters using holistic methods • = • Clinical Process + Clinical Outcome Measure Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

18. Case and Checklist Development • Iterative Process • Involves test committees • Focus on clinical presentation • Checklists “limited” • Encounters with SPs • Appropriateness of content and difficulty • Stepwise progression through pilot and calibration stages • Validation / refinement via review of examinee performance data Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

19. Patient Note • Holistic scoring • Rater training • General policies • Case specific: • Consensus development on key features • Calibration phase • Quality Assurance Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

20. Derivation of Communication / Interpersonal Skills (CIS) Scale • Initial instrument – validated ECFMG scale • Reviewed against national consensus recommendations and commonly used scales Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

21. CIS Subscales • Data gathering skills • Open-ended questions, transitional statements, not interrupting the patient • Information sharing skills • Responsiveness to patient questions/concerns, provision of counseling when appropriate, avoidance of jargon • Personal manner and rapport • Expression of interest in the impact of the illness, concern for patient comfort and modesty Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

22. Step 2 CS: Scoring • Reporting is Pass or Fail only • Examinees must pass all three subcomponents • Feedback to examinees • Performance report – overall and subcomponent outcomes • Failing examinees only – graphical profiles • Intended to show relative strengths and weaknesses Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

23. Sample PerformanceReport Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

24. Sample Performance Profile Lower Performance Borderline Performance Higher Performance Integrated Clinical Encounter Data Gathering Patient Note Communication / Interpersonal Skiills Spoken English Proficiency XXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXX Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

25. Threats to Validity and Reliability • Content and/or tasks not relevant or realistic • Individual test forms vary in content coverage • Scoring methods not appropriate for skills tested • Inconsistency in SP portrayal and scoring • Between cases, across sites, over time • Level of difficulty of cases / exams inconsistent • Standard setting approach appropriate Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

26. Ensuring Fairness and Consistency • Rigorous SP (and SP trainer) training • Meticulous attention to quality assurance • Application of equating procedures Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

27. Standardized Patients Rigorous training – generic and case specific Electronic delivery (“Ecase”) of case materials Sign off process for SPs – Criteria : # portrayals, tests SP Trainers Adherence to training protocols “Training academy” for SP trainers SP and SP Trainer Training Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

28. Quality Assurance • Begins with the SP sign-off process • Monitoring procedures and analyses: • Qualitative (portrayal and scoring accuracy): • Live and video review of SP performances • Quantitative • Score-based analyses • Case level and item level comparisons Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

29. Equating Procedures • Within site • SP-case combination • Between site • Central video review Data gathering Communication and Interpersonal Skills • Patient note rater – case combination Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

30. Standard Setting • USMLE Standard Setting System • Committee assigned responsibility to establish and monitor standards • Decision-making process • Survey of constituent opinion • Standard-setting exercises by independent groups • Examinee performance data and score reliability Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

31. Conclusions • Numerous logistical and psychometric challenges were identified and responses described • In order to ensure exam fairness and consistency: • Sound test and case development practices • Intensive SP and SP trainer training • Rigorous quality assurance • Well-considered equating and standard setting procedures • Supported by ongoing research on reliability and validity Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

32. Speaker Contact Information Richard E. Hawkins, MD National Board of Medical Examiners 3750 Market Street Philadelphia, PA 19104 215-590-9204 / Fax 215-590-9440 Email: rhawkins@nbme.org Websites: http://www.usmle.org http://www.nbme.org Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona