An introduction to multiple choice questions (1-best answer)

1. An introduction to multiple choice questions (1-best answer) Sue Fowell, School of Medicine s.l.fowell@liv.ac.uk

2. Structure • Basic assessment theory • Objective testing methods • MCQs • Writing guidelines • How to identify ‘good’ questions

3. First things first.. • What is the purpose of assessment: • Formative or summative? • What do you want to assess? • Learning objectives • Curriculum map or blueprint • MCQs are not the only format

4. MCQ format Question stem: Who was the oldest member of The Beatles?: Options a. Paul McCartney b. George Harrison c. Ringo Starr d. John Lennon Options a, b and d are distractors

5. Good Assessments • Reliability Does the assessment provide an accurate measurement ? • Validity Does the assessment measure what we think it measures?

6. Facets of assessment • Reliability • Validity • Feasibility • Cost • Acceptability Rigour Practicality

7. Question writing and design Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: The way to a man’s heart is through his:

8. GOLDEN RULE of MCQs* It should be possible to answer the question from looking at the stem alone. *a.k.a the cover-up test

9. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Cancerous growths in the oesophagus most...

10. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: To evaluate whether a MCQ discriminates ...

11. To evaluate whether a MCQ discriminates between candidates of differing ability levels, you should calculate the value of: • The KR-20 • The difficulty index • The discrimination index • The standard error of measurement

12. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll.

13. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Question 1, The deficiency disease resul...

14. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Question n, The pellagra-protective fact...

15. Question 1The deficiency disease resulting from a lack of nicotinic acid in the diet is: • Beriberi • Pellagra • Scurvy • Xerophthalmia

16. Question 2The pellagra-protective factor in the vitamin B complex is: • Biotin • Niacin (nicotinic acid) • Pantothenic acid • Pyridoxine

17. Format • Single best answer; 4 or 5 options • Long stem – short, equal length options • Avoid grammatical clues in stem • some options start with a vowel • not all options follow from the stem • plurals in stem/options • Avoid negatives • NOT • EXCEPT • Present options in a ‘logical’ order • Use ‘homologous’ options • Don’t use ‘all/none of the above’ as options

18. Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance

20. Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance • Avoid imprecise terms e.g. • frequently • usually • never

21. From Case: The use of imprecise terms in examination questions: how frequent is frequently? Academic Medicine. 1994:69(suppl) S4-S6

22. Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance • Avoid imprecise terms e.g. • frequently • usually • never • Apply the ‘cover up’ rule

23. General advice • Write in groups if possible • Brain storm ideas • Review each other’s questions • Ask colleague to review questions WITHOUT answers • Cover-up rule • Use knowledge of typical errors to select distractors • Review question performance

24. Evaluating tests:How to identify ‘good’ questions: Item analysis • difficulty index • performance of ‘distractors’ • discrimination index

25. Difficulty Index (p value) • What proportion of students answer the question correctly?

28. Which distractors do students select? • Similar proportions of students give answer A and C (correct answer) • May need to review content?

29. What is wrong with this question? • Almost all students answer incorrectly • Nearly all students select the same incorrect answer • Incorrectly coded question?

30. Discrimination index

31. Discrimination index Looks at number of students in top and bottom third of class overall who answer this item correctly DI = (Upper 1/3) – (Bottom 1/3) Half total no. of students in 2 groups Ideally, DI is greater than 0.35 = 81.4 – 41.9 100 = 0.395

32. Alternative measure of discrimination

33. What’s wrong with this question?

34. How to calculate discrimination index by hand • Rank students by overall score (descending) • Divide in to equal thirds • Count number of students in upper third that answer item correctly (n1) • Count number of students in lower third that answer item correctly (n2) • DI = (n1 – n2)/(Half total no. of students in 2 groups)

35. Interpretation • Discrimination between students who perform well on test and those that perform less well • For a ‘traditional’ summative assessment, >0.35 • If zero then no difference between these groups • If negative, look at the question carefully! • (Other measure exist – e.g. Point B biserial)

36. References • Bloom, B.S. Taxonomy of Educational Objectives, Handbook 1, The Cognitive Domain. 1956, New York: McKay. • Armstrong, W.G., F.B. Reed, and P.J. Wells, Pitfalls in MCQ design. In: Self-assessment in biochemistry for medicine and dentistry. 1984, Blackwell Scientific: Oxford. p. xiv-xix. • Holsgrove, G. and M. Elzubeir, Imprecise terms in UK multiple-choice questions: what examiners think they mean. Medical Education, 1998. 32(4): p. 343-350.

37. Useful Websites • Case, S.M. and D.B. Swanson, Constructing written test questions for the basic and clinical sciences. 1996: National Board of Medical Examiners. Can be downloaded from: www.nbme.org as ‘Item writing manual’ under ‘publications’