Writing Effective Test Questions for the Basic and Clinical Sciences

1. Writing Effective Test Questions for the Basic and Clinical Sciences Carolyn L. Cambor, M.D. Department of Pathology Jennifer R. Kogan, M.D. Department of Medicine FAPD Series September 29, 2010 Materials adapted from the National Board of Medical Examiners

2. Objectives • Describe the item types currently used by the NBME & the rationale for their use • Describe the steps in writing A-type items • Basic Sciences • Clinical Sciences • Identify technical flaws in test items & understand the importance of avoiding them • Advance your skills in writing A-type test items • If time permits, describe and practice writing R-type test items

3. Steps in Test Development Test purpose Testing time and method of administration Test standardization Test content Number of items Item format Develop items Item selection and evaluation

4. Why learn how to write test items? Ensure that your items are: • Clear & understandable • Test what you want to test • Fair • Provide reproducible results Avoid technical “flaws” • Create confusion • Add unnecessary difficulty • May aid the “testwise student”

5. “Alphabet Soup” of Item Types True-false • X (simple true/false) • K (complex true/false) “1,2,3”, “1,3”, “2,4” “all” • C (A/B/Both/Neither) In general, not recommended; not used in Mod1,2 If used, must be clear, unambiguous with options that are 100% T or 100% F One-best answer • A (4 or more options) • B (4 or 5 option matching sets in sets of 2–5 items) • R (extended matching items in sets of 2-20 items)

6. A-Type Item: Components • Stem Longest part of the item Sufficient information to answer the lead-in question. • Lead-in The question being asked • Options The correct answer and 3-4 distractors

7. A-type Item Stem:A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg. Lead-in:Which of the following muscles is most likely to have been injured? Options:A. Gluteus maximus* Answer B. Gluteus minimus C. Hamstrings D. Iliopsoas E. Obturator internus Distractors

8. Good, A-Type Items Have a Distinctive Shape Stem Lead In Option A (distractor) Option B (answer) Option C (distractor) Option D (distractor) Option E (distractor)

9. Poorly shaped A-type item Short stem & lead-in • Long option • Long option • Long option • Long option

10. 6 Rules for Writing A-type ItemsBasic Sciences 1) Focus on important topics. Ideally based on your objectives. 2)Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

11. 6 Rules for Writing A-type Items:Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

12. 6 Rules for Writing A-type Items:Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

13. Example – Recall or Application? What area of the brain is supplied by blood from the posterior inferior cerebellar artery? A) Location 1 B) Location 2 C) Location 3 D) Location 4 Recall item: rote memory of isolated fact

14. Example – Recall or Application? A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D Application of knowledge: Reach a conclusion, make a prediction, select a course of action

15. 6 Rules for Writing A-type Items:Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

16. Cover the Options Rule Stem Lead in Options a-e

17. Example – Cover the Options A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D

18. Example – Cover the Options A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D

19. Writing the Stem Avoid “Which of the following statements is correct?” “Each of the following statements is correct EXCEPT…” Because Unfocused stems Don’t follow “cover the options” rule Heterogeneous options mix of epidemiology, genetics, mechanisms etc

20. Poorly worded stem: • True statements about cystic fibrosis (CF) include: 1. The incidence of CF is 1:2000 2. Children with CF usually die in their teens. 3. Males with CF are sterile 4. CF is an autosomal recessive disease

21. Patient vignettes are good for “stems” Patient vignettes (stem) • brief for basic sciences • should not require patient care expertise • age gender site of care presenting complaint duration • +/- family history • +/- results of diagnostic tests; initial treatment, etc • material needs to be taught with clinical relevance Lead in • Clearly focused question

22. Patient Vignette Templates: Basic Sciences • A (patient description) has a (type of injury, location). Which of the following structures is most likely to be affected? • A (patient description) has (signs & symptoms). These observations suggest that the disease is a result of the (absence or presence) of which of the following (enzymes, mechanisms)? • A (patient description) dies of (disease). Which of the following is the most likely finding on autopsy?

23. Patient Vignette Templates: Basic Sciences • Which of the following is the most likely mechanism of the therapeutic effect of this (drug class) in patients with (disease)? • (time period) after a (event, such as a trip or meal w/ certain foods) a (patient or group description) became ill with (symptoms and signs). Which of the following (organisms, agents) is most likely to be found on analysis of (food)?

24. During an operation, the arterial PCO2, and pH of an anesthetized patient are monitored. The patient is being ventilated by a mechanical respirator, and the initial values are normal (PCO2=40 mm Hg; pH=7.42). If the ventilation is decreased, which of the following is most likely to occur? Arterial pCO2 A. decrease B. decrease C. decrease D. increase * E. increase F. increase pH A. decrease B. increase C. no change D. decrease * E. increase F. no change Physiology: Patient Vignette

25. At a banquet, the menu included fried chicken, homefried potatoes, peas, chocolate eclairs and coffee. Within 2 hours most of the diners became violently ill, with nausea, vomiting and abdominal pain. Analysis of the contaminated food is most likely to yield large numbers of which of the following organisms? A. Escherechia coliB. Proteus mirabilisC. Salmonella typhimuriumD. Staphylococcus aureus*E. Streptococcus faecalis Microbiology: Patient Vignette

26. Lab Vignettes are good for Basic Sciences “Lab”-vignettes • laboratory examples • research examples • biochemical pathways • drug metabolism

27. Biochemistry: Lab Vignette A B C D II III IV I V E IV In the branched metabolic pathway, a different single enzyme catalyzes each of the individual steps. The enzyme that would be expected to be most severely inhibited by compound V is enzyme: A. A B. B * C. C D. D E. E

28. Pharmacology: Lab Vignette Drug Y has a volume of distribution of 75 L in both younger and older adult men. In younger adults, it has a clearance rate of 15L / h, 50% of which is via the liver and 50% via the kidneys. For younger men, the maintenance regimen is 100 mg every 6 hours. Which of the following regimens will produce essentially the same steady-state concentration in an older man, whose creatinine clearance is half that of younger men, but whose hepatic function is unimpaired? A. 75 mg every 3 hrs B. 75 mg every 6 hrs * C. 75 mg every 9 hours D. 100 mg every 3 hours E. 100 mg every 6 hours F. 100 mg every 12 hours

29. Basic Sciences: Integrated Items Physiology / Pharmacology Microbiology / Pharmacology Pathology / Pharmacology Pathology / Pathophysiology • Stem describes one component, options ask about another • 1 stem is used for 2+ questions, on different topics • Guidelines: • Team preparation • Avoid “cueing” • Avoid “hinging”

30. Integrated Items - Example An unresponsive 58 year old woman is brought to the ED after collapsing at a local shopping center. More history. Physical signs describe a neurologic deficit. 1. The dilated, unreactive left pupil is most consistent with injury to the left List of anatomic structures 2. The extensor posturing on the right is most consistent with injury to the left List of anatomic structures 3. Which of the following herniation syndromes is most consistent with her clinical presentation? List of herniation syndromes

31. Patient Vignettes: Clinical Sciences Include age, gender site of care presenting complaint duration patient history physical findings +/- diagnostic studies +/- initial treatment Stems should not be completely based on real patients include reference material when it would be realistic in practice not use the patient’s or doctor’s own words not include patients who lie

32. Non-vignettes in the Clinical Sciences The most likely renal abnormality in children with nephrotic syndrome and normal renal function is A. Acute poststreptococcal glomerulonephritis B. Hemolytic-uremic syndrome C. Minimal change disease D. Focal and segmental glomerulosclerosis E. Schonlein-Henoch purpura

33. Short Vignette A 2 year old boy has a 1 week history of edema. His blood pressure is 100/60 mmHg and there is generalized edema and ascites. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL. UA shows 4+ protein and no blood. The most likely diagnosis is

34. Long Vignette A 2 year old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mmHg, pulse 110/min respirations 28/min. Exam shows swelling of eyes, abdominal distention and a positive fluid wave. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL.UA shows 4+ protein and no blood. The most likely diagnosis is

35. Writing the Lead-In: Clinical Sciences Diagnosis • Which of the following is the most likely diagnosis? • Which of the following is the most appropriate next step in diagnosis? Management • Which of the following is the most appropriate next step in patient care? Mechanisms of disease • Which of the following is the most likely pathogen? • Which of the following is the most likely explanation for the findings?

36. 6 Rules for Writing A-type Questions • Focus on important topics. Ideally based on course / activity objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. • Assess application of knowledge, not recall. • Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5)Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

37. Writing the Options • Homogeneous in content • Grammatically consistent with stem • Similar in construction and length • Alphabetized or logically ordered • Incorrect/inferior to the correct answer • choice affects question difficulty • Plausible/attractive to uninformed • no distracter should be obviously incorrect • Each should be chosen by some

38. Easy Distractors Who was the primary author of the Declaration of Independence? A. Abraham Lincoln B. Thomas Jefferson* C. Franklin Roosevelt D. King George II E. Catherine the Great

39. More Difficult Distractors Who was the primary author of the Declaration of Independence? A. George Washington B. Thomas Jefferson* C. Alexander Hamilton D. Benjamin Franklin E. James Madison

40. 6 Rules for Writing A-type Questions 1) Focus on important topics. Ideally based on course / activity objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

41. Technical Item Flaws Issues related to Irrelevant Difficulty Issues Related to “Testwiseness”

42. Assess Your “Testwiseness”

43. Testwise Analysis • The primary purpose of the stam is to remove the • carm • denton • menace • stam bar

44. Testwise Analysis • The primary purpose of the stam is to remove the • carm • denton • menace • stam bar “Word repeats” in stem and answer

45. Testwise Analysis • Which of the following pairs has won the greatest number of Abby awards? • Jones and Smith • Smith and Taylor • Smith and White • White and Allen

46. Testwise Analysis • Which of the following pairs has won the greatest number of Abby awards? • Jones and Smith • Smith and Taylor • Smith and White • White and Allen Convergence: correct answer has the most in common with other choices

47. Testwise Analysis • How many pounds of pressure are exerted by a callam? • 2.6 • 150 • 260 • 2600

48. Testwise Analysis • How many pounds of pressure are exerted by a callam? • 2.6 • 150 • 260 • 2600 Convergence: Correct answer has the most in common with other choices

49. Testwise Analysis • The stanon is aided by a • instel • immon • octal • port

50. Testwise Analysis • The stanon is aided by a • instel • immon • octal • port Grammatical cues: only “port” follows grammatically from the stem