370 likes | 550 Views
An introduction to multiple choice questions (1-best answer). Sue Fowell, School of Medicine s.l.fowell@liv.ac.uk. Structure. Basic assessment theory Objective testing methods MCQs Writing guidelines How to identify ‘good’ questions. First things first. What is the purpose of assessment:
E N D
An introduction to multiple choice questions (1-best answer) Sue Fowell, School of Medicine s.l.fowell@liv.ac.uk
Structure • Basic assessment theory • Objective testing methods • MCQs • Writing guidelines • How to identify ‘good’ questions
First things first.. • What is the purpose of assessment: • Formative or summative? • What do you want to assess? • Learning objectives • Curriculum map or blueprint • MCQs are not the only format
MCQ format Question stem: Who was the oldest member of The Beatles?: Options a. Paul McCartney b. George Harrison c. Ringo Starr d. John Lennon Options a, b and d are distractors
Good Assessments • Reliability Does the assessment provide an accurate measurement ? • Validity Does the assessment measure what we think it measures?
Facets of assessment • Reliability • Validity • Feasibility • Cost • Acceptability Rigour Practicality
Question writing and design Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: The way to a man’s heart is through his:
GOLDEN RULE of MCQs* It should be possible to answer the question from looking at the stem alone. *a.k.a the cover-up test
Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Cancerous growths in the oesophagus most...
Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: To evaluate whether a MCQ discriminates ...
To evaluate whether a MCQ discriminates between candidates of differing ability levels, you should calculate the value of: • The KR-20 • The difficulty index • The discrimination index • The standard error of measurement
Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll.
Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Question 1, The deficiency disease resul...
Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Question n, The pellagra-protective fact...
Question 1The deficiency disease resulting from a lack of nicotinic acid in the diet is: • Beriberi • Pellagra • Scurvy • Xerophthalmia
Question 2The pellagra-protective factor in the vitamin B complex is: • Biotin • Niacin (nicotinic acid) • Pantothenic acid • Pyridoxine
Format • Single best answer; 4 or 5 options • Long stem – short, equal length options • Avoid grammatical clues in stem • some options start with a vowel • not all options follow from the stem • plurals in stem/options • Avoid negatives • NOT • EXCEPT • Present options in a ‘logical’ order • Use ‘homologous’ options • Don’t use ‘all/none of the above’ as options
Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance
Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance • Avoid imprecise terms e.g. • frequently • usually • never
From Case: The use of imprecise terms in examination questions: how frequent is frequently? Academic Medicine. 1994:69(suppl) S4-S6
Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance • Avoid imprecise terms e.g. • frequently • usually • never • Apply the ‘cover up’ rule
General advice • Write in groups if possible • Brain storm ideas • Review each other’s questions • Ask colleague to review questions WITHOUT answers • Cover-up rule • Use knowledge of typical errors to select distractors • Review question performance
Evaluating tests:How to identify ‘good’ questions: Item analysis • difficulty index • performance of ‘distractors’ • discrimination index
Difficulty Index (p value) • What proportion of students answer the question correctly?
Which distractors do students select? • Similar proportions of students give answer A and C (correct answer) • May need to review content?
What is wrong with this question? • Almost all students answer incorrectly • Nearly all students select the same incorrect answer • Incorrectly coded question?
Discrimination index Looks at number of students in top and bottom third of class overall who answer this item correctly DI = (Upper 1/3) – (Bottom 1/3) Half total no. of students in 2 groups Ideally, DI is greater than 0.35 = 81.4 – 41.9 100 = 0.395
How to calculate discrimination index by hand • Rank students by overall score (descending) • Divide in to equal thirds • Count number of students in upper third that answer item correctly (n1) • Count number of students in lower third that answer item correctly (n2) • DI = (n1 – n2)/(Half total no. of students in 2 groups)
Interpretation • Discrimination between students who perform well on test and those that perform less well • For a ‘traditional’ summative assessment, >0.35 • If zero then no difference between these groups • If negative, look at the question carefully! • (Other measure exist – e.g. Point B biserial)
References • Bloom, B.S. Taxonomy of Educational Objectives, Handbook 1, The Cognitive Domain. 1956, New York: McKay. • Armstrong, W.G., F.B. Reed, and P.J. Wells, Pitfalls in MCQ design. In: Self-assessment in biochemistry for medicine and dentistry. 1984, Blackwell Scientific: Oxford. p. xiv-xix. • Holsgrove, G. and M. Elzubeir, Imprecise terms in UK multiple-choice questions: what examiners think they mean. Medical Education, 1998. 32(4): p. 343-350.
Useful Websites • Case, S.M. and D.B. Swanson, Constructing written test questions for the basic and clinical sciences. 1996: National Board of Medical Examiners. Can be downloaded from: www.nbme.org as ‘Item writing manual’ under ‘publications’