1 / 9

Blind double marking

Explore the impact of blind double marking on the reliability of essay exams in dental education. Comparison of traditional double marking and blind double marking, statistical analysis, strategies for improving inter-marker agreement, and actions taken for enhancement.

melba
Download Presentation

Blind double marking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Blind double marking Aim: to raise some issues contingent on the introduction of blind double marking of essay exams

  2. BChD (Bachelor of Dental Surgery) • 5 year course • non-modular (curriculum largely determined by General Dental Council) • degree exams in 1st, 2nd, 3rd and 5th year plus ‘Progress to Finals’

  3. The written exam in ‘Finals’ • takes place in June of 5th year • = 40% of marks for Finals • 2 three-hour papers • 4 compulsory essay questions in each paper • each essay marked out of 25 • traditionally double-marked • first double-marked ‘blind’ in 1999

  4. Double-marking vs.blind double-marking • traditionally 2nd marker agreed with 1st mark 70% - 80% of time and disagreements nearly always only 1 - 2 marks (out of 25) • marking blind, absolute agreement dropped to between 48% and 17% with, in one question, 35% of marks being >2 marks apart

  5. Extent of agreement between markers in 1999 final written exam

  6. Analysis of markers’ consensus process(all questions, 48 students)

  7. Inappropriate use of correlation statistics • Correlation statistics show association, not agreement e.g. correlation would be perfect if all marks given by 2nd marker are exactly 10 points higher than those given by 1st marker • therefore use kappa, which measures absolute agreement with reference to the frequency with which marks would be expected to agree by chance Note: agreement can be evaluated as: Poor (<0.20), Fair (0.21 - 0.40), Moderate (0.41 - 0.60), Good (0.61 - 0.80) or Very Good (0.81 - 1.00)

  8. Standard ways of improving inter-marker reliability • re-visit assessment criteria/marking scheme • have standardisation meetings with markers • downgrade importance of essays in assessment and use more objective methods • continue to monitor marking

  9. Action taken between June and November 1999 • Issues raised and discussed at Undergraduate Dental Education Committee and in Dental School newsletter ‘Teaching Quality Matters’ • Assessment criteria reviewed, re-formatted and published in Nathan Bodington • Markers for Operative Dentistry prize exam (December 1999) given verbal briefing about assessment criteria • Result = a marked improvement in reliability !! (but….)

More Related