1 / 12

Principles of Good Assessment and Standard Setting

Principles of Good Assessment and Standard Setting. Katharine Boursicot Reader in Medical Education St George’s, University of London. Principles. Examine what is being assessed Review methods/ blueprinting Consider if formats suited to needs Standard setting.

tiernan
Download Presentation

Principles of Good Assessment and Standard Setting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Principles of Good Assessment and Standard Setting Katharine Boursicot Reader in Medical Education St George’s, University of London

  2. Principles • Examine what is being assessed • Review methods/ blueprinting • Consider if formats suited to needs • Standard setting

  3. Critical questions in assessment • WHY are they doing the assessment? • WHAT are they assessing? • HOW are they assessing it? • HOW WELL is the assessment working?

  4. 1. WHY are they doing the assessment? • Is its purpose: • Formative? • Summative?

  5. 2. WHAT are they testing? • Elements of competence • Knowledge • factual • applied: clinical reasoning • Skills • communication • clinical • Attitudes • professional behaviour Tomorrow’s Doctors, GMC 2003

  6. Performance assessment in vivo: Video, WBA eg mini-CEX, DOPs Does Performance assessment in vitro: OSCEs Knows how (Clinical) Context based tests: SBAs, EMQs, SAQ Knows Factual tests: SBAs Shows how 3. How are they doing the assessment? Test formats Does Shows how Knows how Knows

  7. 4. HOW WELL is the assessment working? Evaluation of assessment systems • Is it valid? • Is it reliable? • Is it doing what it is supposed to be doing? • To answer these questions, we have to consider the characteristics of assessment instruments

  8. Utility function • U = Utility • R = Reliability • V = Validity • E = Educational impact • A = Acceptability • C = Cost • W = Weight U = wrR x wvV x weE x waA x wcC

  9. 100% 0% In-training assessment EMQ/MCQ Utility function • U = Utility • R = Reliability • V = Validity • E = Educational impact • A = Acceptability • C = Cost • w = Weight Weight U = wrR x wvV x weE

  10. 100% Overall assessment programme 0% Individual instrument Utility function • U = Utility • R = Reliability • V = Validity • E = Educational impact • A = Acceptability • C = Cost Weight U = wrR x wvV x weE

  11. Setting the pass mark: characteristics of credible standards • Are they using a recognised method? • Evidence.... • How are the standards applied? The method has to be: • defensible • credible • supported by a body of evidence in the literature • feasible • acceptable to all stakeholders • Norcini, J. J. (2003). Setting standards on educational tests. Medical Education, 37, 464-469.

  12. Principles of Assessment • There is no perfect assessment: compromise is always required • The compromise depends on the context of the assessment • TheQuality of assessments is a matter of the integral assessment programme, rather than of the individual instruments

More Related