Writing Effective Test Questions for the Basic and Clinical Sciences

1. Writing Effective Test Questions for the Basic and Clinical Sciences Carolyn L. Cambor, M.D. Department of Pathology Jennifer R. Kogan, M.D. Department of Medicine FAPD Series September 28, 2011 Materials adapted from the National Board of Medical Examiners

2. Objectives • Describe the item types currently used by the NBME & the rationale for their use • Describe the steps in writing A-type items • Basic Sciences • Clinical Sciences • Identify technical flaws in test items & understand the importance of avoiding them • Advance your skills in writing A-type test items • If time permits, describe and practice writing R-type test items

3. Steps in Test Development Test purpose Testing time and method of administration Test standardization Test content Number of items Item format Develop items Item selection and evaluation

4. Why learn how to write test items? Ensure that your items are: • Clear & understandable • Test what you want to test • Fair • Provide reproducible results Avoid technical “flaws” • Create confusion • Add unnecessary difficulty

5. “Alphabet Soup” of Item Types True-false • X (simple true/false) • K (complex true/false) “1,2,3”, “1,3”, “2,4” “all” • C (A/B/Both/Neither) In general, not recommended; not used in Mod1,2 If used, must be clear, unambiguous with options that are 100% T or 100% F One-best answer • A (4 or more options) • B (4 or 5 option matching sets in sets of 2–5 items) • R (extended matching items in sets of 2-20 items)

6. A-Type Item: Components • Stem Longest part of the item Sufficient information to answer the lead-in question. • Lead-in The question being asked • Options The correct answer and 3-4 distractors

7. A-type Item Stem:A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg. Lead-in:Which of the following muscles is most likely to have been injured? Options:A. Gluteus maximus* Answer B. Gluteus minimus C. Hamstrings D. Iliopsoas E. Obturator internus Distractors

8. Good, A-Type Items Have a Distinctive Shape Stem Lead In Option A (distractor) Option B (answer) Option C (distractor) Option D (distractor) Option E (distractor)

9. Poorly shaped A-type item Short stem & lead-in • Long option • Long option • Long option • Long option

10. 6 Rules for Writing A-type ItemsBasic Sciences 1) Focus on important topics. Ideally based on your objectives. 2)Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

11. 6 Rules for Writing A-type Items:Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

12. 6 Rules for Writing A-type Items:Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

13. Example – Recall or Application? What area of the brain is supplied by blood from the posterior inferior cerebellar artery? A) Location 1 B) Location 2 C) Location 3 D) Location 4 Recall item: rote memory of isolated fact

14. Example – Recall or Application? A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D Application of knowledge: Reach a conclusion, make a prediction, select a course of action

15. 6 Rules for Writing A-type Items:Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

16. Writing the Stem Avoid “Which of the following statements is correct?” “Each of the following statements is correct EXCEPT…” Because it will result in Unfocused stems, insufficient information Heterogeneous options mix of epidemiology, genetics, mechanisms etc And The item won’t follow “cover the options” rule

17. Cover the Options Rule Stem Lead in Options a-e

18. Example 1: Well written or not? • Which of the following is true about cystic fibrosis (CF): A. The incidence of CF is 1:200 live births B. The median life expectancy is 21 years. C. Males with CF have normal fertility D. CF is an autosomal recessive disease

19. Example 1: Not well written • Which of the following is true about cystic fibrosis (CF): A. The incidence of CF is 1:200 live births B. The median life expectancy is 21 years. C. Males with CF have normal fertility D. CF is an autosomal recessive disease

20. Example 2: Well written or not? A 32-year-old man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 37.8 C (100 F), blood pressure is 130/80 mm Hg, pulse is 94/min, and respirations are 42/min and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis

21. Example 2: Yes well written A 32-year-old man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 37.8 C (100 F), blood pressure is 130/80 mm Hg, pulse is 94/min, and respirations are 42/min and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis

22. Patient vignettes are good for “stems” Patient vignettes (stem) • brief for basic sciences • should not require patient care expertise • age gender site of care presenting complaint duration • +/- family history • +/- results of diagnostic tests; initial treatment, etc • material needs to be taught with clinical relevance Lead in • Clearly focused question

23. Patient Vignette Templates: Basic Sciences • A (patient description) has a (type of injury, location). Which of the following structures is most likely to be affected? • A (patient description) has (signs & symptoms). These observations suggest that the disease is a result of the (absence or presence) of which of the following (enzymes, mechanisms)? • A (patient description) dies of (disease). Which of the following is the most likely finding on autopsy?

24. Patient Vignette Templates: Basic Sciences • Which of the following is the most likely mechanism of the therapeutic effect of this (drug class) in patients with (disease)? • (time period) after a (event, such as a trip or meal w/ certain foods) a (patient or group description) became ill with (symptoms and signs). Which of the following (organisms, agents) is most likely to be found on analysis of (food)?

25. During an operation, the arterial PCO2, and pH of an anesthetized patient are monitored. The patient is being ventilated by a mechanical respirator, and the initial values are normal (PCO2=40 mm Hg; pH=7.42). If the ventilation is decreased, which of the following is most likely to occur? Arterial pCO2 A. decrease B. decrease C. decrease D. increase * E. increase F. increase pH A. decrease B. increase C. no change D. decrease * E. increase F. no change Physiology: Patient Vignette

26. At a banquet, the menu included fried chicken, homefried potatoes, peas, chocolate eclairs and coffee. Within 2 hours most of the diners became violently ill, with nausea, vomiting and abdominal pain. Analysis of the contaminated food is most likely to yield large numbers of which of the following organisms? A. Escherechia coliB. Proteus mirabilisC. Salmonella typhimuriumD. Staphylococcus aureus*E. Streptococcus faecalis Microbiology: Patient Vignette

27. Lab Vignettes are good for Basic Sciences “Lab”-vignettes • laboratory examples • research examples • biochemical pathways • drug metabolism

28. Biochemistry: Lab Vignette A B C D II III IV I V E IV In the branched metabolic pathway, a different single enzyme catalyzes each of the individual steps. The enzyme that would be expected to be most severely inhibited by compound V is enzyme: A. A B. B * C. C D. D E. E

29. Pharmacology: Lab Vignette Drug Y has a volume of distribution of 75 L in both younger and older adult men. In younger adults, it has a clearance rate of 15L / h, 50% of which is via the liver and 50% via the kidneys. For younger men, the maintenance regimen is 100 mg every 6 hours. Which of the following regimens will produce essentially the same steady-state concentration in an older man, whose creatinine clearance is half that of younger men, but whose hepatic function is unimpaired? A. 75 mg every 3 hrs B. 75 mg every 6 hrs * C. 75 mg every 9 hours D. 100 mg every 3 hours E. 100 mg every 6 hours F. 100 mg every 12 hours

30. Basic Sciences: Integrated Items Common Combinations: Physiology / Pharmacology Microbiology / Pharmacology Pathology / Pharmacology Pathology / Pathophysiology

31. Integrated Items: Case Clusters

32. Case Clusters • MCQs associated with the same patient presentation • Format • Brief case followed by several (i.e. 3) MCQs. • Each question addresses different aspect of the case from different perspective • Often difficult for single faculty member to prepare b/c items draw from several basic science disciplines • Team approach useful for preparation

33. Case Cluster Example • A 34-year-old HIV positive woman has had severe watery diarrhea for the past four days. Two months earlier she had infectious mononucleosis. She abuses drugs intravenously. Physical examination shows dehydration and marked muscle weakness.

34. Question 1 Laboratory studies are most likely to show A. increased serum K+ concentration B. decreased serum Ca2+ concentration C. increased serum HCO3- concentration *D. increased serum Na+ concentration E. increased serum pH

35. Question 2 In evaluating the cause of the diarrhea, which of the following is most appropriate? A. Colonic biopsy to identify Giardia lamblia B. Culture of the oral cavity for Candida albicans C. Duodenal biopsy to identify Entamoeba histolytica D. Gastric aspirate to identify Mycobacterium avium-intracellulare *E. Stool specimen to identify Cryptosporidium

36. Question 3 Further studies to evaluate her HIV infection show the ratio of helper T lymphocytes to suppressor T lymphocytes to be 0.3. This occurs because HIV A. induces proliferation of helper T lymphocytes B. induces proliferation of suppressor T lymphocytes *C. infects cells with CD4 receptors D. infects macrophages E. stimulates the synthesis of leukotriene

37. Avoid Cueing • Providing hints to the answers of earlier questions in later questions. • Cluster question example: Patient with chest pain • 1st question: most likely cause of CP • 2nd question: select most appropriate drug rx • Testwise examinee can rule out diagnoses (and drugs) simply by comparing the option lists. • Each diagnosis in 1st question needs matching drug in 2nd (and vise-versa)

38. Avoid Hinging • Creating questions where students must know the answer to one question in order to answer other questions • Exception: Topic tested is so important that the item writer willing to have students receive either all or none of the points associated with a cluster

39. Example An unresponsive 58-year-old woman is brought to the ED after collapsing while shopping. Her family reports she felt well that morning but developed a progressive headache while shopping. She has HTN and atrial fibrillation and is taking a BP medication and warfarin. Her BP is 220/130 mm Hg and her respiratory pattern is one of apnea alternating with hyperpnea. She responds only to noxious stimuli with extensor posturing involving the R arm and leg. Fundoscopic examination reveals papilledema involving the L optic disc. Pupils are 3.0/7.0 (R/L) with no reaction to light on the L. There is a L gaze preference. There is diffuse hyperreflexia (R > L) and Babinski’s sign is present bilaterally.

40. Question 1 The dilated, unreactive left pupil is most consistent with injury to the left A. optic nerve B. optic tract *C. oculomotor nerve D. lateral geniculate nucleus E. superior colliculus

41. Question 2 The extensor posturing on the right is most consistent with injury to the left A. telencephalon B. diencephalon *C. midbrain D. pons E. medulla

42. Question 3 Her respiratory pattern is best described as A. normal *B. Cheyne-Stokes C. central neurogenic hyperventilation D. apneustic E. ataxic

43. Question 4 (hinged question) Which of the following herniation syndromes is most consistent with her clinical presentation? A. Cingulate gyrus beneath the falx *B. Temporal lobe uncus across the tentorium C. Diencephalon through the tentorial notch D. Brain stem through the tentorial notch E. Cerebellar tonsils through the foramen magnum

44. Patient Vignettes: Clinical Sciences Include age, gender site of care presenting complaint duration patient history physical findings +/- diagnostic studies +/- initial treatment Stems should not be completely based on real patients include reference material when it would be realistic in practice not use the patient’s or doctor’s own words not include patients who lie

45. Non-vignettes in the Clinical Sciences The most likely renal abnormality in children with nephrotic syndrome and normal renal function is A. Acute poststreptococcal glomerulonephritis B. Hemolytic-uremic syndrome C. Minimal change disease D. Focal and segmental glomerulosclerosis E. Schonlein-Henoch purpura

46. Short Vignette A 2 year old boy has a 1 week history of edema. His blood pressure is 100/60 mmHg and there is generalized edema and ascites. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL. UA shows 4+ protein and no blood. The most likely diagnosis is

47. Long Vignette A 2 year old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mmHg, pulse 110/min respirations 28/min. Exam shows swelling of eyes, abdominal distention and a positive fluid wave. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL.UA shows 4+ protein and no blood. The most likely diagnosis is

48. Writing the Lead-In: Clinical Sciences Diagnosis • Which of the following is the most likely diagnosis? • Which of the following is the most appropriate next step in diagnosis? Management • Which of the following is the most appropriate next step in patient care? Mechanisms of disease • Which of the following is the most likely pathogen? • Which of the following is the most likely explanation for the findings?

49. 6 Rules for Writing A-type Questions • Focus on important topics. Ideally based on course / activity objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. • Assess application of knowledge, not recall. • Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5)Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty

50. Writing the Options • Homogeneous in content • Grammatically consistent with stem • Similar in construction and length • Alphabetized or logically ordered • Incorrect/inferior to the correct answer • choice affects question difficulty • Plausible/attractive to uninformed • no distracter should be obviously incorrect • Each should be chosen by some