Writing Multiple Choice Questions

1. Writing Multiple Choice Questions Amy S. Oxentenko, MD, FACP, FACG, AGAF Mayo Clinic, Rochester

2. Disclosures • Relevant Relationships: • Consulting work for MKSAP 14-17 • On the IM-ITE exam committee

3. Outline/Objectives • Define thetest taker level before writing • List components of a multiple-choicequestion • Objective, stem, lead line, options • Annotated description of answers • Evaluate your questions

4. Who Are You Writing For? • Med school GI pathophysiology course? • More details; physiology on USMLEs • Resident end-of-rotation exam? • Presentation, testing, management • GI Fellow board review course? • More advanced understanding of above • IM or GI CME course? • Very general vs very specific

5. What are the Stakes of the Questions? • Board certification? • Passing a class/rotation? • Simple knowledge assessment? • Solely for interaction?

6. The Stakes Reproducible Accurate Learning

7. What Should Be Tested? • For a clerkship/CME course • Match goals/objectives • For a standardized exam • Match the targeted learner level

8. How Often Will Learner Encounter Information • Frequent • More relevant to test often • Atypical manifestations • Infrequent • Test the “can’t miss” things • Life threatening • Typical manifestations

9. Types of Questions • Single best answer • With clinical scenario • Factoid questions • All of the following EXCEPT • Matching • True/False • A and B, A and C, all of the above

10. What are the Components of a Multiple Choice Question (MCQ)?

11. Question Components STEM LEAD LINE OPTIONS

12. Question Components ANSWER OBJECTIVE EXPLANATION

13. PLEASE TAKE THE PRETEST.

14. Steps to Writing the MCQ • Determine the objective • Create the lead line and options • Draft the stem • Write the explanation, if required

15. The Objective

16. Example: Objective • Recall alcoholic hepatitis What is wrong with this?

17. Example: Objective • Treat a patient with alcoholic hepatitis with corticosteroids in the absence of contraindications More focused

18. The Objective • Succinct sentence • Specifies a clinical skill to be learned • Uses action verbs to delineate a goal • Recognize, diagnose, treat, manage • Avoid vague action verbs • Remember, recall, know • Allows you to match curriculum

19. VERB+TOPIC+ MODIFIER Objective Format

20. Action Verbs for Objectives

21. Examples of Objectives:The Good, the Bad, the Ugly • Good: • Diagnose niacin deficiency in a patient with carcinoid syndrome • Bad: • Recall features of niacin deficiency • Ugly: • Understand niacin deficiency

22. Other Objective Examples • Select appropriate testing needed before beginning a biologic agent on an IBD patient. • Evaluate for splenic vein thrombosis in a patient with isolated gastric varices. • Manage a patient with celiac disease who has persistent symptoms on a GFD. • Identify peptic ulcers with stigmata for rebleeding that require endoscopic therapy.

23. Lead Line

24. Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic, with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen. The next step would be to administer • Niacin • CT scan of the head • Thiamine • Insulin • Dismissal to home What is wrong with this?

25. Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic, with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen. What is the next best step in management? • Niacin • CT scan of the head • Thiamine • Insulin • Dismissal to home Keep lead line general; avoid syntax errors

26. Lead Line • Most commonly used lead lines: • Which of the following is the most likely diagnosis? • What is the next best step in management? • Half treatment options, half diagnostic options • Avoid negative lead lines: • All of the following EXCEPT • Which of the following is LEAST likely • Which would you NOT do

27. Lead Line • Keep the lead line generic • “Next step” or “management” • Allows diagnostic & therapeutic options • Avoid writing syntax or grammatical errors that help exclude answers • “Cover the options” rule • Should be able to guess the answers

28. The Options

29. Example #1: Options What is the next best step in management? • Add IV metronidazole • Continue povancomycin • Start cholestyramine • Begin rifaximin • Stop the povancomycin What is wrong with this?

30. Example #1: Options What is the next best step in management? • Add IV metronidazole • Continue povancomycin • Start cholestyramine • Begin rifaximin • Stop the povancomycin Mutually exclusive

31. Example #2: Options What is the next best step in management? • Metronidazole 500 mg po TID x 14 days • Vancomycin 250 mg IV QID x 10 days • Metronidazole 500 mg IV QID x 14 days • Vancomycin 125 mg po QID x 14 days What is wrong with this?

32. Example #2: Options What is the next best step in management? • Metronidazole 500 mg po TID x 14 days • Vancomycin 250 mg IV QID x 10 days • Metronidazole 500 mg IV QID x 14 days • Vancomycin 125 mg po QID x 14 days Testing too many concepts!Drugs, dose, route, duration

33. Example #3: Options The cause of diarrhea in most patients with Zollinger-Ellison syndrome is: • Antacid use • High glucagon levels • Fat malabsorption from pancreatic lipase inactivation by acid production • Bacterial overgrowth What is wrong with this?

34. Example #3: Options The cause of diarrhea in most patients with Zollinger-Ellison syndrome is: • Antacid use • High glucagon levels • Fat malabsorption from pancreatic lipase inactivation by acid production • Bacterial overgrowth Correct answer longer, detailed

35. The Options • 4 or 5 options (less may be better) • One is not right and the rest 100% wrong C E A B D Wrong Right

36. The Options • 4 or 5 options (less may be better) • One is not right and the rest 100% wrong • One is correct, the rest plausible C E A B D Least correct Most correct

37. Options • Avoid mutually exclusive options • Increases/decreases/same • Always/never; stop/continue • Each option should test one concept • Drug OR dose OR route OR duration • Keep options brief; similar in length • Options should be homogeneous • Length, complexity

38. Stem

39. The Stem • Chronologic order of presentation • Age, gender (avoid race unless needed) • Site of visit (ER, clinic, hospital) • Chief complaint (add features) • PMH/Meds/FH/SH (relevant or distracter) • Vitals/Exam/Labs/Tests (pertinent)

40. The Stem • Fine to have details to make the other options attractive; avoid extraneous info • It takes time to read; make sure it is fitting in with time to administer test • Organize labs if many

41. The Explanation

42. The Explanation • Explain why right answer correct first • Explain why wrong answers incorrect • Do this chronologically • Example for explanation order: • If correct answer is B • B, A, C, D, E

43. How Do You Analyze Your Questions?

44. Question Performance • Goal for correct answer 50-70% • 40-90% may be allowed at times • Difficulty score of > 90% (too easy) • Difficulty score of < 40% (too hard) • See if it discriminates: • Top test takers • Bottom test takers

45. Performs well; difficulty ok and discriminating

46. Performs poor; difficulty ok, non-discriminating

47. Performs poor; difficulty low (hard), non-discriminating

48. http://www.nbme.org/publications/item-writing-manual-download.htmlhttp://www.nbme.org/publications/item-writing-manual-download.html

50. The Pretest: How Did You Do?