Performance-based standard setting methods

1. Performance-based standard setting methods Katharine Boursicot Trudie Roberts

2. Performance-based standard setting methods • Borderline group method • Contrasting group method • Regression based standard method Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, van der Vleuten C Comparison of a rational and an empirical standard setting procedure for an OSCE, Medical Education, 2003 Vol 37 Issue 2, Page 132 Kaufman DM, Mann KV, Muijtjens AMM, van der Vleuten CPM. A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Acad Med 2000; 75:267-271.

4. The examiner’s role in standard setting • Uses the examiner’s clinical expertise to judge the candidate’s performance • Examiner allocates a global judgement based on the candidate’s performance at that station • Remember thelevelof the examination Pass Borderline Fail

5. Borderline Group Method Test score distribution Checklist 1. Hs shjs sjnhss sjhs sjs sj 2. Ksks sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk dd 6. Hskl;s skj sls ska ak akl ald 7. Hdhhddh shs ahhakk as TOTAL     Borderline score distribution  Pass, Fail, Borderline Passing score

6. Contrasting groups method Test score distribution Checklist 1. Hs shjs sjnhss sjhs sjs sj 2. Ksks sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk dd 6. Hskl;s skj sls ska ak akl ald 7. Hdhhddh shs ahhakk as TOTAL    Fail Pass   Pass, Fail, Borderline P/B/F Passing score

7. 1 2 3 4 5 Regression based standard Checklist X= passing score 1. Hs shjs sjnhss sjhs sjs sj 2. Ksks sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk dd 6. Hskl;s skj sls ska ak akl ald 7. Hdhhddh shs ahhakk as TOTAL  Checklist Score  X    Overall rating 1 2 3 4 5 Clear Borderline Clear Excellent Outstanding fail pass 1 = Clear fail 2 = Borderline 3 = Clear pass 4 = Excellent 5 = Outstanding

8. Performance-based standard setting • Advantages • Utilises the expertise of the examiners • they are observing the performance of the students • they are in a position to make a (global) judgement about the performance • based on • their clinical expertise • expected standards for the level of the test • knowledge of the curriculum/teaching

9. Borderline/ contrasting/ regression-based methods • Advantages • Large number of examiners set a collective standard while observing the candidates – not just an academic exercise • Reliable: cut-off score based on large sample of judgments • Credible and defensible: based on expert judgment in direct observation

10. Performance-based standard setting • Disadvantages • Requires large cohort of candidates to achieve enough numbers in the ‘borderline’ group • Passing score not known in advance • Judgments not independent of checklist scoring • Requires expert processing of marks immediately after the exam • Checking of results • Delay in producing results