380 likes | 571 Views
Standard setting Determining the pass mark. The Old Way. …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna I can see dividing the bottom from the rest. Assessment. Blueprinting Assessments Pass or Fail? Methods of Standard Setting
E N D
Standard setting Determining the pass mark
The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna I can see dividing the bottom from the rest
Assessment • Blueprinting Assessments • Pass or Fail? Methods of Standard Setting • - Angoff • - Borderline • - Hofstee • …………………………………………………………. • Assessment of Professional Behaviours
Vision Statement The University of Sheffield strives to produce excellent medical graduates. The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university. The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice. The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems. The instructional approach will consist of a spine of problem, case and patient-based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas. Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities. Assessment, both formative and summative, will be closely matched to defined outcomes. The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support. A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement. Excellent graduates Outcome focussed Integrated programme Student choice Undifferentiated problems Integrated Learning Activities Able learners Assessment matches outcomes Curriculum management Quality enhancement
Summary • Assessments should match local/national outcomes for Validity • Multiple observations required for Reliability • Electronic data collection for optimal Feasibility • Good information to students and staff to enhance Acceptance
Pass Competent Fail Incompetent
Pass measurement error Fail
Standard setting Knowledge Skills
Standard setting Angoff Hofstee Knowledge Borderline method Skills
D I S C U S S +/- M O D I F Y Angoff (modified) What is the mark that a student who just passes would score for each item? Mean mark for all items is the pass mark Written and clinical examinations
Angoff The pass mark for this exam using a modified Angoff is 63.7%
Angoff standard setting • Determine the pass mark for the ‘just-competent’ student • and write it in the box • Do not assume that the pass mark is 50% or 60% • or constant
Borderline procedure Used for clinical (OSCE) examinations
Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations
Station 15 (static) Oral Lesions Pass/Fail A A 1 50 A 2 A 3 40 A 30 Count A A 20 A A 10 A A A A A A A A A A A A A A A A 0 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 Score
pass Borderline score 58.41 Borderline group fail
pass Borderline score 58.41 Borderline group How wide should this band be? fail
pass Borderline score 58.41 Borderline group How wide should this band be? fail +/- 1 - standard deviation - standard error - or what?
The Standard Error of Measurement • depends on the reliability of the test (R) • depends on the standard deviation of the • test (SD) • SEM = SD 1 - R • acts as a confidence interval in high • stakes situations
The Standard Error of Measurement • depends on the reliability of the test (R) • depends on the standard deviation of the • test (SD) • SEM = SD 1 - R • acts as a confidence interval in high • stakes situations R = Variance Variance x Error
Pass Score 60.93 Pass } + 1 SEM Borderline score 58.41 Borderline group }- 1 SEM Fail Fail score 55.88
Internal Reliability of the ExamCronbach’s Alpha Item / Stations Population (students) Split half Correlations (SH) Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together
Grading System Top Mark Distinction Good Pass Pass Pass Pass Mark Borderline Borderline Mark Fail Borderline Fail
Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance
Hofstee procedure Used for written examinations Judges review a copy of the whole exam Judges indicate: minimum % failure rate maximum % failure rate minimum acceptable cut percentage maximum acceptable cut percentage Administer test, plot curve and read off standard
100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %
100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %
100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %
100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %
100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %
100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %