Constructing flawless MCQs for assessment and learning

1. Constructing flawless MCQs for assessment and learning Jack W. Strandhoy, PhD

2. “Students can escape from bad teaching, but not from poor assessment” • (Boyd, 1995)

3. Integrated Components of Teaching and Learning

4. How can we improve our MCQs? • Purpose: formative, summative • Focus: important concepts, common, clear-cut, or potentially catastrophic clinical problems • Format: question type, no technical flaws • Level: strive to test application of knowledge, higher order thinking

5. From D. DaRosa

6. Fundamental knowledge item Which of the following dephosphorylates AChE? • Phenoxybenzamine • Phenylephrine • Physostigmine • Pralidoxime * • Prazosin

7. Applied basic science mechanism item Reactivation of cholinesterase shortly after toxic exposure to an OP insecticide can be accomplished with: • Phenoxybenzamine • Phenylephrine • Physostigmine • Pralidoxime * • Prazosin

8. Clinical task item You are treating a farm worker in the ED who was accidentally poisoned with an OP. You treat parasym. symptoms with atropine. Which drug do you choose to reactivate cholinesterase? • Phenoxybenzamine • Phenylephrine • Physostigmine • Pralidoxime * • Prazosin

9. Integrating information A 13 year-old boy is brought to the ED 5 days after a camping trip. He has a petechial rash on his arms and legs that began on the wrists and ankles and spread in the last two days. He has a headache, malaise, fever and chills. A skin biopsy and Ab titer are positive for a rickettsial infection. He is prescribed an antibiotic that acts by which of the following mechanisms? • Acts as a detergent on the rickettsial cell wall • Binds at the 30S ribosome to prevent attachment of tRNA • Depolymerizes microtubules in the cytosol • Inhibits DNA gyrase • Prevents ergosterol biosynthesis Note: Info from several disciplines but question is focused on one topic.

10. Other formats • Video clips, sound files, radiographic images, virtual microscopy • Much easier with computerized exams than on paper • Can test higher order thinking • Being incorporated into NBME exams

11. Type A one-best-answer items • Focus on important concepts; common, clear-cut, or catastrophic clinical problem • Test application of knowledge rather than factual recall • Question answer should be clear and answerable from just the stem • Distractorsshould be homogeneous • Avoid technical flaws

12. Which of the following pairs has won the most Abby awards? A. Jones & Smith B. Smith & Taylor C. Smith & White D. White & Allen Questions with structural flaws

13. Which of the following pairs has won the most Abby awards? A. Jones & Smith B. Smith & Taylor C. Smith & White D. White & Allen Ans = C Questions with structural flaws

14. Which of the following pairs has won the most Abby awards? A. Jones & Smith B. Smith & Taylor C. Smith &White D.White& Allen Ans = C Which of the following drug combinations is most likely to produce ototoxicity? A. tetracycline & HCTZ B. gentamicin & furosemide C. ampicillin & furosemide D. gentamicin & triamterene Ans = B Questions with structural flaws

15. Questions with structural flaws • Grammatical clues – a and an • Longest answer is correct • Absolutes – always, never [usu. wrong] • Vague, “weasel words” – usually, often, frequently • Negatives, double negatives, “except” or “not” statements

16. Questions with structural flaws • Answers too long or complicated • Word repeats as clues • Inconsistent, awkward structure • Terms like “none of the above”

17. Interpretation of imprecise terms NBME, 2003

18. Questions with structural flaws • Stem should provide all of the information necessary to answer the question • Flawed: • Which of the following is true about pseudogout? • Use longer stems and shorter options rather than the other way around

19. Elementary School Quiz • True/False: • A shape that has 4 sides is called a square • An apple is a sphere • The numerical value of the term π is 3.1415926585

20. Medical School Quiz • True/False: • The incidence of cystic fibrosis is 1:2000 • Children with CF usually die in their teens • Males with CF are sterile

21. Be aware: = multiple T / F All of the following concerning H. pylori and peptic ulcer dz (PUD)are correct EXCEPT: • Infection w/ this organism is seen in the majority of pts with PUD • Infection enhances gastric acid secretion • Only a minority of infected pts develop PUD • The organism secretes urease that breaks down urea into NH3 and CO2, which forms the basis of a diagnostic test for the presence of the organism.

22. Type A: One best answer • Which of the following most approximates a sphere? • Apple • Banana • Frisbee • Hula hoop • Pretzel

23. Which of the following most approximates a sphere? • Apple • Basketball • Egg • Charlie Brown’s head • Watermelon

24. Type R questions:Extended Matching • Theme: broad or narrow • Options list: homogeneous, alphabetical. Words or short phrases • Lead in statement: match how? • At least two stems (small vignettes). Make similar in structure within a set • Make sure there’s only 1 best answer and at least 4 distractors/stem

25. gentamicin G. penicillin V • hydrochlorothiazide H. prednisone • lithium carbonate I. primaquine • metronidazole J. vancomycin • allopurinol F. morpine sulfate For each pt. description, select the MOST likely drug that was administered. A 9 yo boy is admitted with gram (+) sepsis. An antibiotic is quickly infused IV. The child complains of a flushed feeling and you note redness around the face and neck. A 35 yo woman psychiatry patient complains of excessive urination and thirst. Urine osmolality was 100 mOsm/Kg water. A 35 yo woman with lupus nephritis is treated with chronic medication. She develops gastric irritation, altered fat deposition to her face and back, and osteoporosis. Type R questions

26. Acetaminophen P. Sulfasalazine • Amiodarone Q. Tetracycline • ACE inhibitors ….. R. Verapamil For each pt, select the drug most likely to have caused adverse effect A 56 yo man with recurrent ventricular arrhythmias began taking an antiarrhythmic drug 5 mo ago. He now has progressive cough, dyspnea and low-grade fever. ESR is increased. CXR shows diffuse interstitial pneumonia. PFT show that diffusing capacity for CO is decreased. {amiodarone} A 62 yo man with COPD begins therapy with an antihypertensive drug. Two wks later, he has marked worsening of dyspnea and clearly audible wheezing. {propranolol} NBME examples, Type R

27. Adjust the difficulty of type R questions with the options e.g., antibiotics vs cephalosporins

28. Available from NBME.org“Constructing Written Test Questions for the Basic and Clinical Sciences” 3rd ed. Susan Case & David Swanson 181 pages

29. Template for clinical vignette style questions • Item stem(patient vignette): describe a patient with a problem • Age, gender; Site of care (ED, etc.) • HPI: Presenting complaint; Duration • Pt. hx; family hx; Physical findings, studies, Dx, initial treatments • Lead-in: e.g., Which of the following is the most likely diagnosis? • Options: List of diagnoses • Long stem, short options better than v/v

30. Additional Item Templates • Which of the following is the most likely mechanism for (drug) in patients with (dz)? • A patient has (signs, symptoms). Which of the following (drugs) is best treatment? Which drug should be avoided? • A patient has (side effects). Which drug is she most likely taking?

31. Integrating information A 13 year-old boy is brought to the ED 5 days after a camping trip. He has a petechial rash on his arms and legs that began on the wrists and ankles and spread in the last two days. He has a headache, malaise, fever and chills. A skin biopsy and Ab titer are positive for a rickettsial infection. He is prescribed an antibiotic that acts by which of the following mechanisms? • Acts as a detergent on the rickettsial cell wall • Binds at the 30S ribosome to prevent attachment of tRNA • Depolymerizes microtubules in the cytosol • Inhibits DNA gyrase • Prevents ergosterol biosynthesis Note: Info from several disciplines but question is focused on one topic.

32. Using Clinical Vignettes No vignette: The most likely renal abnormality in children with nephrotic syndrome and normal renal function is: A. Acute poststreptococcal glomerulonephritis B. Hemolytic-uremic syndrome C. Minimal change nephrotic syndrome* D. Nephrotic syndrome due to focal and segmental glomerulosclerosis E. Schönlein-Henoch purpura with nephritis

33. Clinical summary vignette: A 2-year-old boy has a 1-week history of edema. Blood pressure is 100/60. There is generalized edema and ascites. Serum creatinine is 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL Urinalysis shows 4+ protein and no blood. The most likely diagnosis is:

34. Clinical observations vignette: A 2-year-old boy has developed swelling of his ankles and around his eyes for the past week. Blood pressure is 100/60, pulse 110/min, and respirations 28/min. In addition to swelling around his eyes and 2+ pitting edema of his ankles, he has abdominal distention with a positive fluid wave. Serum creatinine is 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL Urinalysis shows 4+ protein and no blood. The most likely diagnosis is:

35. Research vignettes • Item stem– lab data, graphs, dose-response curves • Lead in– • Most likely interpretation? • Next logical experiment? • Best statistical analysis of the data? • Best control experiment? • Options -

36. Guidelines for good questions • Focus on important concepts; don’t waste time testing trivial facts • Match exam question to objective in content and application; encourage higher order thinking • Screen for technical flaws, ambiguities • Examine poor question stats for reasons • Explore multidisciplinary vignettes and questions; require data analysis for answers

37. Guidelines for good questions • You should be able to put your hand over the answers, read the stem, answer it based on the information, then choose the answer from the list. • Good questions should test more than just knowledge – integration, synthesis, judgement.