1 / 32

“USMLE Step 2 Clinical Skills”

“USMLE Step 2 Clinical Skills”. Richard E. Hawkins, MD National Board of Medical Examiners. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona. Presentation Structure . Overview of USMLE Step 2 CS Measures to ensure fairness and consistency.

brooke-levy
Download Presentation

“USMLE Step 2 Clinical Skills”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. “USMLE Step 2 Clinical Skills” Richard E. Hawkins, MD National Board of Medical Examiners Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  2. Presentation Structure • Overview of USMLE Step 2 CS • Measures to ensure fairness and consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  3. Overview of USMLE Step 2 CS Purpose and Delivery Model Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  4. USMLE: Purpose • Three step examination system primarily designed to support the medical licensing process • Jointly sponsored by • Federation of State Medical Boards (FSMB) • National Board of Medical Examiners (NBME) • Step 2 CS: collaboration with ECFMG Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  5. USMLE: Purpose • Overall purpose • Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  6. USMLE: Purpose • Overall purpose • Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills that constitute the basis of safe and effective patient care Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  7. USMLE: Purpose • Patient-centered Skills • History taking / Physical examination • Communication and Interpersonal Skills • Medical Record Documentation • Important for safe and effective patient care • History/PE  Diagnosis and Management • Communication  Patient Health Outcomes • Medical Record  Errors and Patient Safety Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  8. USMLE Step 2 CS • Enhancement to USMLE Step 2 • USMLE Step 2 – components: • Clinical knowledge (CK) • Clinical skills (CS) • Standard for Step 2 – appropriate for entry into postgraduate training Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  9. USMLE Step 2 CS: Logistics • Regional delivery model • Optimal combination of convenience, cost-efficiency and standardization • Five regional test centers across US • Projected examine volume – 30,000+ / year • Individual center capacity • 3 examinations / day (33 examinees); up to 7 days/week Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  10. Requirements of a High StakesPerformance-based Examination Ensuring Fairness and Consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  11. Threats to Validity and Reliability • Content and/or tasks not relevant or realistic • Individual test forms vary in content coverage • Scoring methods not appropriate for skills tested • Inconsistency in SP portrayal and scoring • Between cases, across sites, over time • Level of difficulty of cases / exams inconsistent • Standard setting approach appropriate Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  12. USMLE Step 2 CS: Content • Step 2 CS Blueprint • Defines content categories • In meeting blueprint specifications, each test form provides: • Adequate sampling of content domain • Comparable content between test forms Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  13. USMLE Step 2 CS: Content Blueprint Content Categories: • Common and important medical problems / patient presentations • Acuity • Age • Gender • Race / ethnicity Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  14. Case Content Cardiovascular Respiratory Gastrointestinal Musculoskeletal Constitutional Neurological Psychiatric Genitourinary Women’s health Unclassified / multi-system Case Acuity Acute Subacute / Chronic Test Form Patient age Age less than 18 Age 18 – 44 Age 45 – 64 Age 65 + Patient Gender Male Female Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  15. Step 2 CS: Structure & Tasks • 12 patient encounters • 15 min. for encounter / 10 min. for patient note • Each encounter: • Elicit pertinent history, Perform appropriate physical examination, Communicate effectively • Document: • Findings from the history and physical • Diagnostic impression / Further work-up Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  16. Step 2 CS: Score Components • Integrated Clinical Encounter (ICE) • Data gathering: history and physical exam • Patient note • Communication / Interpersonal Skills (CIS) • Gathering information; sharing information; manner & rapport • Spoken English Proficiency (SEP) • Listener effort, examinee pronunciation / word choice Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  17. Scoring Approach: ICE • Data gathering (Hx / PE) • Dichotomous checklists completed by SPs • + • Patient note • Physician raters using holistic methods • = • Clinical Process + Clinical Outcome Measure Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  18. Case and Checklist Development • Iterative Process • Involves test committees • Focus on clinical presentation • Checklists “limited” • Encounters with SPs • Appropriateness of content and difficulty • Stepwise progression through pilot and calibration stages • Validation / refinement via review of examinee performance data Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  19. Patient Note • Holistic scoring • Rater training • General policies • Case specific: • Consensus development on key features • Calibration phase • Quality Assurance Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  20. Derivation of Communication / Interpersonal Skills (CIS) Scale • Initial instrument – validated ECFMG scale • Reviewed against national consensus recommendations and commonly used scales Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  21. CIS Subscales • Data gathering skills • Open-ended questions, transitional statements, not interrupting the patient • Information sharing skills • Responsiveness to patient questions/concerns, provision of counseling when appropriate, avoidance of jargon • Personal manner and rapport • Expression of interest in the impact of the illness, concern for patient comfort and modesty Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  22. Step 2 CS: Scoring • Reporting is Pass or Fail only • Examinees must pass all three subcomponents • Feedback to examinees • Performance report – overall and subcomponent outcomes • Failing examinees only – graphical profiles • Intended to show relative strengths and weaknesses Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  23. Sample PerformanceReport Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  24. Sample Performance Profile Lower Performance Borderline Performance Higher Performance Integrated Clinical Encounter Data Gathering Patient Note Communication / Interpersonal Skiills Spoken English Proficiency XXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXX Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  25. Threats to Validity and Reliability • Content and/or tasks not relevant or realistic • Individual test forms vary in content coverage • Scoring methods not appropriate for skills tested • Inconsistency in SP portrayal and scoring • Between cases, across sites, over time • Level of difficulty of cases / exams inconsistent • Standard setting approach appropriate Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  26. Ensuring Fairness and Consistency • Rigorous SP (and SP trainer) training • Meticulous attention to quality assurance • Application of equating procedures Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  27. Standardized Patients Rigorous training – generic and case specific Electronic delivery (“Ecase”) of case materials Sign off process for SPs – Criteria : # portrayals, tests SP Trainers Adherence to training protocols “Training academy” for SP trainers SP and SP Trainer Training Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  28. Quality Assurance • Begins with the SP sign-off process • Monitoring procedures and analyses: • Qualitative (portrayal and scoring accuracy): • Live and video review of SP performances • Quantitative • Score-based analyses • Case level and item level comparisons Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  29. Equating Procedures • Within site • SP-case combination • Between site • Central video review Data gathering Communication and Interpersonal Skills • Patient note rater – case combination Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  30. Standard Setting • USMLE Standard Setting System • Committee assigned responsibility to establish and monitor standards • Decision-making process • Survey of constituent opinion • Standard-setting exercises by independent groups • Examinee performance data and score reliability Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  31. Conclusions • Numerous logistical and psychometric challenges were identified and responses described • In order to ensure exam fairness and consistency: • Sound test and case development practices • Intensive SP and SP trainer training • Rigorous quality assurance • Well-considered equating and standard setting procedures • Supported by ongoing research on reliability and validity Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

  32. Speaker Contact Information Richard E. Hawkins, MD National Board of Medical Examiners 3750 Market Street Philadelphia, PA 19104 215-590-9204 / Fax 215-590-9440 Email: rhawkins@nbme.org Websites: http://www.usmle.org http://www.nbme.org Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

More Related