Standard Setting for a Performance-Based Examination for Medical Licensure

1. Standard Setting for a Performance-Based Examination for Medical Licensure Sydney M. Smee Medical Council of Canada Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

2. MCC Qualifying Examination Part II • OSCE format - 12 short stations • 5 or 10 minutes per patient encounter • Physicians observe and score performance • Required for medical licensure in Canada • Prerequisites • Passed MCCQE Part I (Knowledge & Clinical reasoning) • Completed 12 months of post-graduate clinical training • Pass/Fail criterion-referenced examination • Multi-site administration - twice per year • Overall fail rate 10%-30% • Implemented 1992 Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

3. Why do it? • Requested by licensing authorities, largely in response to two issues: • Increase in complaints, many centered around communication skills. • Public accountability - OSCE to serve as an “audit” of training of all candidates seeking licensure in Canada. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

4. Blueprint Considerations • Four domains *History-taking *Patient Interaction *Physical Examination *Management • Multi-disciplinary / multi-system content • Patient demographics • Two formats *5+5 couplets & 10 minute • Each case based on an MCC Objective Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

5. Standard for MCCQE Part II • Acceptably competent for entry to independent practice • Conjunctive standard • Pass by total score AND • Pass by minimum number of stations • High performance in a few stations does not compensate for overall poor performance • Just passing enough stations does not compensate for overall poor performance Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

6. Translating a Standard to a Pass Mark • Pilot exam: Ebel method • Items rated for relevance and importance • Pass based on most relevant and important items • Failed 40% • First two administrations: Angoff method • Estimated score for the minimally competent candidate • Pass based on average of estimates per instrument • Pass marks varied more than the test committee liked • Test committee did not like the task • 1994: Adopted borderline group method Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

7. Physicians as Scorers Three Assumptions: • Clinicians do not require training to judge candidate behaviour according to checklists for basic clinical skills • Most clinicians can make expert judgments about candidate performance • Being judged by clinicians is vital for a high-stakes examination Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

8. Physicians as Standard Setters Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

9. Global Rating Question • Did the candidate respond satisfactorily to the needs/problems presented by this patient? • Borderline Unsatisfactory • Unsatisfactory • Inferior • Borderline Satisfactory • Good • Excellent Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

10. Numbers.... • 1,000-2,200 candidates per administration • Examiners each observe 16-32 candidates • 20-60 examiners per case • Number of candidates identified as borderline per case ranges from 150-500 • Collect >99% of data for global rating item Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

11. Modified Borderline Group Method • Examiners (content experts) identify borderline candidates based on the 6-point scale • Scores of borderline candidates define performance that “describes” the pass standard • Examiner judgments are translated into a pass mark by taking the mean score for the borderline candidates for each case Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

12. Pass Marks by Case Across Exams • Challenge to assess pass marks over multiple administrations • Scoring instruments are revised post-exam • Rating scale items have been revised • Rating scale items have been added to cases • As competency and difficulty of cases changes, so do cut scores Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

13. Setting Total Exam Pass Mark • Pass marks for cases are summed • Add one standard error of measure (3.2% ) • Pass mark falls between 1 to 1.5 SD below mean score • Station performance is reviewed by Central Examination Committee • Then the standard for the number of stations passed is set • Standard has been 8/12 since 2000 Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

14. Outcomes • 15,331 candidates became eligible in 2000 – 2005 • 6,099 have yet to attempt MCCQE Part II • 8,514 have passed • 718 or 7.7% failed • 2,243 candidates were eligible prior to 2000 and also took MCCQE Part II in 2000 – 2005 • 2,166 have passed • 77 or 3.4% failed and are likely out of the system • Fail rates do not reflect impact on repeat takers • Focused hundreds of candidates on remediation Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

15. Limitation • Current approach is easy to implement but it relies upon • Large number of standard setters per case • Large number of test takers in borderline group • Smaller numbers would lead to more effort • Increase training of examiners • Impose stricter selection criteria on standard setters Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

16. What’s ahead? • Increasing number of candidates to be assessed each year • Modifications to the administration are needed • Predictive validity study currently in progress • Use non-physician examiners? • Which type of cases, who sets standard? • Add more administrations? • Case development / challenge of piloting content Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

17. Medical Council of Canada Ottawa Sydney M. Smee, M.Ed. Manager, MCCQE Part II www.mcc.ca Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona