Standard setting Determining the pass mark

1. Standard setting Determining the pass mark

2. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna I can see dividing the bottom from the rest

3. Assessment • Blueprinting Assessments • Pass or Fail? Methods of Standard Setting • - Angoff • - Borderline • - Hofstee • …………………………………………………………. • Assessment of Professional Behaviours

4. Vision Statement The University of Sheffield strives to produce excellent medical graduates. The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university. The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice. The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems. The instructional approach will consist of a spine of problem, case and patient-based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas. Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities. Assessment, both formative and summative, will be closely matched to defined outcomes. The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support. A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement. Excellent graduates Outcome focussed Integrated programme Student choice Undifferentiated problems Integrated Learning Activities Able learners Assessment matches outcomes Curriculum management Quality enhancement

5. Align ObjectivesTeaching & learning Assessment

6. Summary • Assessments should match local/national outcomes for Validity • Multiple observations required for Reliability • Electronic data collection for optimal Feasibility • Good information to students and staff to enhance Acceptance

7. Pass Competent Fail Incompetent

8. Pass measurement error Fail

9. Standard setting Knowledge Skills

10. Standard setting Angoff Hofstee Knowledge Borderline method Skills

11. D I S C U S S +/- M O D I F Y Angoff (modified) What is the mark that a student who just passes would score for each item? Mean mark for all items is the pass mark Written and clinical examinations

12. Angoff The pass mark for this exam using a modified Angoff is 63.7%

13. Angoff standard setting • Determine the pass mark for the ‘just-competent’ student • and write it in the box • Do not assume that the pass mark is 50% or 60% • or constant

14. Borderline procedure Used for clinical (OSCE) examinations

16. Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations

17. Borderline Method

18. Station 15 (static) Oral Lesions Pass/Fail A A 1 50 A 2 A 3 40 A 30 Count A A 20 A A 10 A A A A A A A A A A A A A A A A 0 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 Score

19. pass Borderline score 58.41 Borderline group fail

20. pass Borderline score 58.41 Borderline group How wide should this band be? fail

21. pass Borderline score 58.41 Borderline group How wide should this band be? fail +/- 1 - standard deviation - standard error - or what?

22. The Standard Error of Measurement • depends on the reliability of the test (R) • depends on the standard deviation of the • test (SD) • SEM = SD 1 - R • acts as a confidence interval in high • stakes situations

23. The Standard Error of Measurement • depends on the reliability of the test (R) • depends on the standard deviation of the • test (SD) • SEM = SD 1 - R • acts as a confidence interval in high • stakes situations R = Variance Variance x Error

24. Pass Score 60.93 Pass } + 1 SEM Borderline score 58.41 Borderline group }- 1 SEM Fail Fail score 55.88

28. Internal Reliability of the ExamCronbach’s Alpha Item / Stations  Population (students)  Split half Correlations (SH) Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together

29. Grading System Top Mark Distinction Good Pass Pass Pass Pass Mark Borderline Borderline Mark Fail Borderline Fail

30. Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance

31. Hofstee procedure Used for written examinations Judges review a copy of the whole exam Judges indicate: minimum % failure rate maximum % failure rate minimum acceptable cut percentage maximum acceptable cut percentage Administer test, plot curve and read off standard

32. 100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %

33. 100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %

34. 100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %

35. 100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %

36. 100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %

37. 100 90 80 70 60 % candidates 50 40 30 20 10 0 50 60 70 80 Student score %