1 / 31

Standard Setting

A standard is a statement about whether an examination performance is good enough for a particular purposeA particular score that serves as the boundary between passing and failingA numerical answer to the question

ivrit
Download Presentation

Standard Setting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Standard Setting Alan Denison

    2. A standard is a statement about whether an examination performance is good enough for a particular purpose A particular score that serves as the boundary between passing and failing A numerical answer to the question “how much is enough?” Definition

    3. Based on judgements about examinees’ performances against a social or educational construct Year 2 medical student vs MRCP Standards

    4. The method must be: Defensible Credible Supported by a body of evidence in the literature Feasible Acceptable to all stakeholders Setting the pass mark…

    5. Relative (norm-referenced) Absolute (criterion-referenced) Types of Standards

    6. Relative (norm-referenced) Based on judgements about groups of test takers A set proportion fail regardless of how well they perform e.g. top 28% pass Most appropriate where requirement exists to control numbers proceeding Types of Standards

    12. Absolute (criterion-referenced) Based on judgements about test questions Based on judgements about the performance of individual examinees Based on how much the examinees know Candidates pass or fail depending on if they meet specified criteria (ie pass mark defined in advance) Types of Standards

    14. Used for standard setting of written exams in MBChB programme Not just for medics! (Ebel/Hofstee/Cohen) (Modified) Angoff Method

    15. Judges selected Discussion Nature and purpose of test Nature of examinees What is adequate/inadequate knowledge Concept of the borderline student Read question item Estimate % of borderline group that would answer correctly Record rating, discuss, change Repeat for each item Passing score = aggregated mean What (should) happen at a standard setting meeting

    16. A – His pet hamster died B – His daughter told him he was embarrassing C – It was his birthday D – His audition on Britain’s got Talent was rated poorly by Simon Cowell E - He noticed his hair was going grey Alan was a little bit glum one day last week. What is the most likely explanation for this?

    20. Total = 3.08 Passing % = 62%

    21. Focus attention on item content Easy to use Lots of evidence supporting their use Used +++ in high stakes tests Advantages

    22. What is a borderline group? Choose your judges wisely A bit tedious for some exams Disadvantages

    23. Standard Setting for OSCEs

    24. Borderline Group Borderline Regression Standard Setting for OSCEs

    25. Global rating made by each examiner Pass / Borderline / Fail Mean score of borderline students used to set pass mark for each station Borderline Group

    27. Easy to calculate Only uses borderline data Need >20 candidates in this category to be reliable Limited quality assurance metrics Borderline Group

    28. Examiner makes judgement at each station Clear pass / pass / borderline / fail / bad fail Borderline Regression

    30. Uses scores from all candidates No minimum number of candidates in borderline group Borderline Regression

    31. Cronbach alpha Generalisability theory Other exam metrics

More Related