1 / 37

An introduction to multiple choice questions (1-best answer)

An introduction to multiple choice questions (1-best answer). Sue Fowell, School of Medicine s.l.fowell@liv.ac.uk. Structure. Basic assessment theory Objective testing methods MCQs Writing guidelines How to identify ‘good’ questions. First things first. What is the purpose of assessment:

beyonce
Download Presentation

An introduction to multiple choice questions (1-best answer)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. An introduction to multiple choice questions (1-best answer) Sue Fowell, School of Medicine s.l.fowell@liv.ac.uk

  2. Structure • Basic assessment theory • Objective testing methods • MCQs • Writing guidelines • How to identify ‘good’ questions

  3. First things first.. • What is the purpose of assessment: • Formative or summative? • What do you want to assess? • Learning objectives • Curriculum map or blueprint • MCQs are not the only format

  4. MCQ format Question stem: Who was the oldest member of The Beatles?: Options a. Paul McCartney b. George Harrison c. Ringo Starr d. John Lennon Options a, b and d are distractors

  5. Good Assessments • Reliability Does the assessment provide an accurate measurement ? • Validity Does the assessment measure what we think it measures?

  6. Facets of assessment • Reliability • Validity • Feasibility • Cost • Acceptability Rigour Practicality

  7. Question writing and design Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: The way to a man’s heart is through his:

  8. GOLDEN RULE of MCQs* It should be possible to answer the question from looking at the stem alone. *a.k.a the cover-up test

  9. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Cancerous growths in the oesophagus most...

  10. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: To evaluate whether a MCQ discriminates ...

  11. To evaluate whether a MCQ discriminates between candidates of differing ability levels, you should calculate the value of: • The KR-20 • The difficulty index • The discrimination index • The standard error of measurement

  12. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll.

  13. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Question 1, The deficiency disease resul...

  14. Don’t forget: You can copy-paste this slide into other presentations, and move or resize the poll. Poll: Question n, The pellagra-protective fact...

  15. Question 1The deficiency disease resulting from a lack of nicotinic acid in the diet is: • Beriberi • Pellagra • Scurvy • Xerophthalmia

  16. Question 2The pellagra-protective factor in the vitamin B complex is: • Biotin • Niacin (nicotinic acid) • Pantothenic acid • Pyridoxine

  17. Format • Single best answer; 4 or 5 options • Long stem – short, equal length options • Avoid grammatical clues in stem • some options start with a vowel • not all options follow from the stem • plurals in stem/options • Avoid negatives • NOT • EXCEPT • Present options in a ‘logical’ order • Use ‘homologous’ options • Don’t use ‘all/none of the above’ as options

  18. Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance

  19. Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance • Avoid imprecise terms e.g. • frequently • usually • never

  20. From Case: The use of imprecise terms in examination questions: how frequent is frequently? Academic Medicine. 1994:69(suppl) S4-S6

  21. Content • Avoid testing trivia • Avoid testing factual recall – test higher-order performance • Avoid imprecise terms e.g. • frequently • usually • never • Apply the ‘cover up’ rule

  22. General advice • Write in groups if possible • Brain storm ideas • Review each other’s questions • Ask colleague to review questions WITHOUT answers • Cover-up rule • Use knowledge of typical errors to select distractors • Review question performance

  23. Evaluating tests:How to identify ‘good’ questions: Item analysis • difficulty index • performance of ‘distractors’ • discrimination index

  24. Difficulty Index (p value) • What proportion of students answer the question correctly?

  25. Which distractors do students select? • Similar proportions of students give answer A and C (correct answer) • May need to review content?

  26. What is wrong with this question? • Almost all students answer incorrectly • Nearly all students select the same incorrect answer • Incorrectly coded question?

  27. Discrimination index

  28. Discrimination index Looks at number of students in top and bottom third of class overall who answer this item correctly DI = (Upper 1/3) – (Bottom 1/3) Half total no. of students in 2 groups Ideally, DI is greater than 0.35 = 81.4 – 41.9 100 = 0.395

  29. Alternative measure of discrimination

  30. What’s wrong with this question?

  31. How to calculate discrimination index by hand • Rank students by overall score (descending) • Divide in to equal thirds • Count number of students in upper third that answer item correctly (n1) • Count number of students in lower third that answer item correctly (n2) • DI = (n1 – n2)/(Half total no. of students in 2 groups)

  32. Interpretation • Discrimination between students who perform well on test and those that perform less well • For a ‘traditional’ summative assessment, >0.35 • If zero then no difference between these groups • If negative, look at the question carefully! • (Other measure exist – e.g. Point B biserial)

  33. References • Bloom, B.S. Taxonomy of Educational Objectives, Handbook 1, The Cognitive Domain. 1956, New York: McKay. • Armstrong, W.G., F.B. Reed, and P.J. Wells, Pitfalls in MCQ design. In: Self-assessment in biochemistry for medicine and dentistry. 1984, Blackwell Scientific: Oxford. p. xiv-xix. • Holsgrove, G. and M. Elzubeir, Imprecise terms in UK multiple-choice questions: what examiners think they mean. Medical Education, 1998. 32(4): p. 343-350.

  34. Useful Websites • Case, S.M. and D.B. Swanson, Constructing written test questions for the basic and clinical sciences. 1996: National Board of Medical Examiners. Can be downloaded from: www.nbme.org as ‘Item writing manual’ under ‘publications’

More Related