Aviad Haramati, PhD Peggy A. Weissinger, EdD Georgetown University School of Medicine

1. Georgetown University Center for New Designs in Learning & Scholarship and Office of Faculty and Academic AffairsWriting Multiple Choice Questions Aviad Haramati, PhD Peggy A. Weissinger, EdD Georgetown University School of Medicine CNDLS & OFAA Georgetown University

2. Agenda • Welcome & introductions • Why do we assess? • Common flaws in MCQ • Guidelines for writing quality MCQ • Let’s do it! • Resources

3. Why do we assess?

4. Why do we assess? Objectives Activities Assessment Adapted from Felder & Brent ( 1991)

5. Why do we assess? What should my students know? What should they be able to do? Objectives Activities Assessment Adapted from Felder & Brent ( 1991)

6. Why do we assess? Objectives What type of activities will promote and develop this knowledge? What type of activities can help students achieve the learning objectives? Activities Assessment Adapted from Felder & Brent ( 1991)

7. Why do we assess? Objectives How will I know that they have achieved the objectives? Activities Assessment Adapted from Felder & Brent ( 1991)

8. Congruence! Objectives Activities Assessment Adapted from Felder & Brent ( 1991)

9. What am I testing? Before creating an exam, ask yourself: • What do the students have to know (or do)? • What am I testing?

10. What am I testing? Core Principles: • Decide exactly what you want to test. • Eliminate as many ambiguities as possible. • Create distractors based firmly on what you want to test. • After it is over, use it as a learning opportunity! • Give feedbackfor your distracters which explains why they’re wrong. • Give feedback for the correct answer explaining why it’s right. Adapted from http://hotpot.uvia.ca/howto/mcquestion.htm TLISI - Georgetown University

11. Technical Flaws Issues Related to Testwiseness • Grammatical cues Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

12. Grammatical Cues A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of • examination of cerebrospinal fluid • glucose with vitamin B1 (thiamine) • CT scan of the head • phenytoin • diazepam Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 19 TLISI - Georgetown University

13. Technical Flaws Issues Related to Testwiseness • Grammatical cues • Logical cues Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

14. Logical Cues Crime is • equally distributed among the social classes • overrepresented among the poor • overrepresented among the middle class and rich • primarily an indication of psychosexual maladjustment • Reaching a plateau of tolerability for the nation a, b, c are collectively exhaustive d, e may be partially correct but not same dimension as others Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 20 TLISI - Georgetown University

15. Technical Flaws Issues Related to Testwiseness • Grammatical cues • Logical cues • Absolute terms • Long correct answer Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

16. Long Correct Answer Secondary gain is • synonymous with malingering • a frequent problem in obsessive-compulsive disorder • a complication of a variety of illnesses and tends to prolong many of them • never seen in organic brain damage. Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 20 TLISI - Georgetown University

17. Technical Flaws Issues Related to Testwiseness • Grammatical cues • Logical cues • Absolute terms • Long correct answer • Word repeats Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

18. Word Repeats A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called • derealization • depersonalization • derailment • focal memory deficit • signal anxiety

19. Technical Flaws Issues Related to Testwiseness • Grammatical cues • Logical cues • Absolute terms • Long correct answer • Word repeats • Convergence strategy Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

20. Convergence Strategy Local anesthetics are most effective in the • anionic form, acting from inside the nerve membrane • cationic form, acting from inside the nerve membrane • cationic form, acting from outside the nerve membrane • uncharged form, acting from inside the nerve membrane • uncharged form, acting from outside the nerve membrane

21. Technical Flaws Issues Related to Irrelevant Difficulty • Options long, complicated, or double Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

22. Long, Complicated Options Peer review committees in HMOs may move to take action against a physician’s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following? • Proper notice, a tribunal empowered to make the decision, a chance to confront witnesses against him/her, and change to present evidence in defense. • Notice, an impartial forum, council, chance to hear and confront evidence against him/her. • Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense. Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 23) TLISI - Georgetown University

23. Technical Flaws Issues Related to Irrelevant Difficulty • Options long, complicated, or double • Numeric data not stated consistently • Terms in options are vague • Language in options not parallel • Options in non-logical order • Stems are tricky or unnecessarily complicated Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

24. Tricky Stems Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are: I: 46, XX, -14, +T (14q21q) pat II: 46, XX, -14, +T (14q21q) de novo III: 46, XX, -14, +T (14q21q) mat IV: 46, XX, -21, +T (14q21q) pat V: 47, XX, -21, +T (21q21q) (parents not karyotyped) • III, IV, I, V, II • IV, III, V, I, II • III, I IV, V, II • IV, III, I, V, II • III, IV, I, II, V Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 25. TLISI - Georgetown University

25. Technical Flaws Issues Related to Irrelevant Difficulty • Options long, complicated, or double • Numeric data not stated consistently • Terms in options are vague • Language in options not parallel • Options in non-logical order • Stems are tricky or unnecessarily complicated • Answer is “hinged” to a related item Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

26. Item Shape Appropriately Shaped Item Long Stem a. b. c. d. e. Poorly Shaped Item Short Stem a. b. c. d. TLISI - Georgetown University e.

27. When Writing Options... …make sure that they are: • Homogenous in content • Incorrect or inferior to the correct answer • Plausible and attractive to the uninformed • Similar to the correct answer in construction • Grammatically consistent and logically compatible with the stem Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

28. Your opinion, please! Osteopenia is caused by which of the following? • Boron • Phosphorus • Magnesium • Strontium

29. Edited Item A 55 year-old woman presents with osteopenia. In reviewing her dietary profile, you suspect one of the following may be deficient. Which is it? A. Boron B. Phosphorus C. Magnesium D. Strontium

30. Your opinion, please! A 60 year-old male runner complains of being confused, irritable and very lethargic. He has hypertension for which he is taking 10 mg of Lisinopril, an angiotensin converting enzyme inhibitor. He states that he drinks ~3L of water and eats “lots of fruits and vegetables” daily. Which of the following is the likely factor producing his symptoms? • Potassium depletion • Sodium depletion • Lisinopril resistance • Decreased renal blood flow

31. Edited Item A 60 year-old male runner complains of being confused, irritable and very lethargic. He has hypertension for which he is taking 10 mg of Lisinopril, an angiotensin converting enzyme inhibitor. He states that he drinks ~3L of water and eats “lots of fruits and vegetables” daily. Which of the following is the likely factor producing his symptoms? • Hypokalemia • Hyponatremia • Hypocalcemia • Hypomagnesemia

32. General Guidelines • Bulk of question in the stem. • Try “cover-the-options” test. • Avoid superfluous information. • Avoid “tricky” and overly complex items. • Write grammatically consistent options. • Avoid absolutes! (e.g. always, never, all) And most important: Focus on important concepts. Don’t waste time testing trivial facts. Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

33. Let’s Do It! • Write one multiple choice question. Reference: Case & Swanson (NBME) Constructing Written Test Questions TLISI - Georgetown University

34. SAMPLE ITEM A 25 year-old medical student who is using natural family planning to chart her fertility, just started her Surgery rotation.  Since the start of the quarter, she has been waking up at 4:00 a.m. every day and getting less sleep than usual, as well as feeling generally stressed.  During her next menstrual cycle, which of the following phases is most likely to be longer than normal for her?(MD) • Follicular Phase • Menstrual Phase • Ovulatory Phase • Secretory Phase

35. SAMPLE ITEM When practicing the Billings Ovulation Method of NFP, infertile days of the cycle include… (MD) • Dry days after menses • Every day from the 4th day after the Peak Day until the end of the cycle • Both A and B • All of the above

36. SAMPLE ITEM Functional Residual Capacity or FRC (AN) • Is best measured by following Peak Flows • Is the equilibrium volume when the tendency of the lung to expand is balanced by the tendency of the chest wall to retract after full inspiration • Is usually greater than the residual volume • Is maximized when the patient is in deep sleep • Cannot be maintained by active inspiration

37. Pulling it together! Summary: • Decide what you want to test. • Well-written objectives make the job easier. • Use guidelines for writing quality MCQ. • Local resources here to help!

38. Contact information Peg Weissinger paw33@georgetown.edu 202-687-4556 Aviad Haramati haramati@georgetown.edu 202-687-1021 CNDLS & OFAA Georgetown University