MCQ’s 3: Aligning with Course Objectives

1. MCQ’s 3:Aligning with Course Objectives

2. MCQ Check-list • This presentation will focus on: • Why we write them • What type is best • What flaws are to be avoided • What content to test • How to construct it

3. Why do we test? • Formative assessment • Communicate to the students what material is important • Identify areas of deficiency in need of remediation or further learning • Summative assessment • Determine final grades • Make promotion decisions • Both • Motivate students to study • Identify areas where the course/curriculum is weak

4. What Should Be Tested? Exam content should match course objectives Important topics should be weighted more heavily than less important topics The testing time devoted to each topic should reflect the relative importance of the topics

5. What Should Be Tested? • Sampling - Performance on the sample provides a basis for estimating achievement in the broader domain that is actually of interest • Questions sample: • TOPICS – eg MCC presentations • SKILLS – eg Course Objectives – determining diagnosis, next step in management, etc.

6. Term 4A, Jan 2012

10. Course Blueprint – ‘Alignment’

11. Types of MCQ’s True/False family • Assess recall of isolated facts • Stems must be clear and unambiguous • Options must be absolutely true or false or examinee must decide how true options are • Generally not recommended – usually have subtle flaws

12. Types of MCQ’s: True/False Which of the following statements is true regarding contraception? oral contraceptives should not be initiated until six weeks after therapeutic abortion to avoid the risk of hypercoagulability an intrauterine device can provide good contraceptive protection 5 years after insertion oral contraceptives should not be prescribed to individuals under the age of 15 without parental consent lesbian women do not require counseling about contraception or sexually transmitted diseases oral contraceptives should not be prescribed for women overthe age of 35

13. Types of MCQ’s Least correct Most correct One-Best-Answer family (A type) Options may be diagrammed: D C A E B Correct answer is the most likely correct Better application of knowledge, integration, synthesis and judgment

14. Types of MCQ’s: One-Best-Answer family A 24 year old university student came to see you because she is missing several days of school each month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender uterosacral nodule on pelvic examination. Which one of the following conditions is most likely responsible for her symptoms? • Chronic pelvic inflammatory disease • Endometriosis • Adenomyosis • Fibroids

15. MCQ Check-list • Why we write them • What type is best • What flaws are to be avoided • What content to test • How to construct it

16. What to test? Items can be classified bycognitive processes required to answer the question: • Recall (Memory)– tests knowledge of isolated facts • Interpretation (Comprehension)– review information to reach conclusion, eg diagnosis • Problem solving (Reasoning) – situation requires action, eg next step in management • Process depends on background of trainee

17. What to test? Simpler classification based on task: • Application of knowledge item ** • Reach conclusions • Make predictions • Select course of action • Recall item • Test knowledge of isolated facts

18. What to test? • Test application of knowledge using clinical vignettes • Adds ‘face-validity’ • Usually selects more important, less trivial • Identifies student who’ve memorized but do not understand BUT experience in patient care should not be necessary

19. What to test - summary Focus item on key concepts and principles that are essential information (without access to references) for all examinees to understand Test material that is relevant to learning in pre-clinical courses and beyond Avoid items that only require recall of isolated facts Avoid esoteric or interesting topics that are not essential

20. What to test - summary • Test application of knowledge using clinical vignettes to pose medical decisions in patient care situations • Focus items on common or potentially catastrophic problems • Pose clinical decisions that would be expected of a successful examinee • ie Avoid clinical situations that would be handled by a (sub)specialist

21. Writing the Questions Construct stem –include all necessary information to arrive at the right answer Choose lead-in – pose a clear question Choose distractors

22. Stem* • Usually a clinical scenario • Clear, unambiguous • Should be long relative to options • Includes all pertinent information • Patient’s age/gender • Clinical setting • Complaint • Other important info – history, physical findings, test results

23. Lead-in* • Follows stem • In the form of a question • Should relate to the stem • Should be answerable without looking at options • “Cover the Options test” • Should clearly communicate the learning objective

24. Distractors • Each should be selected by some, therefore all plausible, none obviously incorrect • 3 or 4 choices • Common misconceptions • Faulty reasoning

25. Distractors Homogeneous in content to the correct answer Should be in same category as correct answer– diagnosis, treatment, diagnostic test Clearly incorrect or inferior to the correct answer Plausible and attractive to the uninformed Similar to the correct answer in construction and length Grammatically consistent and logically compatible with the stem

26. General Guidelines for Item Construction • Include as much of the item as possible in the stems: Long stems, short options • Avoid negatively phrased items (except or not in the lead-in). If you must, use only with short options • Avoid writing questions of the form: • Which of the following statements is correct? • Each of the following statements is correct EXCEPT • Unfocussed • Heterogeneous options

27. “Top-down” versus “Bottom-up” Top-Down Knowledge or recall question Cite a disease and then ask what patient findings are expected Structured like textbooks Clinically backward

28. “Top-down” versus “Bottom-up” Bottom-up Application of knowledge Gives findings and asks examinee to indicate the disease Examinees need to be able to synthesize information from several pages of a textbook (or notes) to answer these questions Clinically realistic

29. Basic Rules for MCQ’s (A type)* Focus item on important concept Item should assess application of knowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover the options’ test All distractors should be homogeneous Avoid technical item flaws

30. Reference Constructing Written Test Questions For the Basic and Clinical Sciences • Case & Swanson, NBME 1996

31. Clinical Presentation A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain and vomiting.

33. Clinical Presentation A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain and vomiting.

34. Clinical presentation A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain and vomiting. This was preceded by a 1 week history of urinary frequency and urgency. On physical examination, VSS are stable and she is afebrile. Abdomen is non-tender. She is noted to have mild Right CVA tenderness.

36. Clinical Presentation 1) A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain and vomiting. This was preceded by a 1 week history of urinary frequency and urgency. On physical examination, VSS are stable and she is afebrile. Abdomen is non-tender. She is noted to have mild Right CVA tenderness. Which of the following investigations would be the next step to make a diagnosis? a) abdominal X-ray b) abdo/pelvic USS c) urinalysis d) CBC

38. Presentation 2) A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain and vomiting. This was preceded by a 1 week history of urinary frequency and urgency. On physical examination, VSS are stable and she is afebrile. Abdomen is non-tender. She is noted to have mild Right CVA tenderness. Urinalysis reveals +++ blood, + WBC and nitrites neg. CBC is normal. Which of the following is the most likely diagnosis? a) appendicitis b) pyelonephritis c) nephrolithiasis d) cystitis

40. Presentation 3) A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain, preceded by urinary frequency and urgency. VSS are stable and she is afebrile. She is noted to have mild Right CVA tenderness. Urinalysis reveals +++ blood, + WBC, nitrites neg. You suspect nephrolithiasis. Which of the following investigations is the best choice to confirm the diagnosis? a) Abdominal and pelvic USS b) abdominal x-ray c) CT scan d) IVP

42. Presentation 4) A 28 year old female presents to the ER with a 2 hour history of severe ride sided abdominal pain with R CVA tenderness. A CT scan confirms the presence of a R UVJ stone measuring approximately 3 mm with complete ureteric obstruction. There are no signs of Urinary Tract infection. You provide analgesia. Which of the following is the next step in managing this problem? a) Hydration b) Percutaneousnephrostomy c) Cystoscopy and basket retrieval of stone d) Lithotripsy

44. Presentation 5) A 28 year old female has recently recovered from her third bout of renal colic. She is otherwise healthy. Which of the following investigations would be most helpful in finding the cause of her recurrent nephrolithiasis? a) IVP b) CT scan c) serum Calcium d) 24 hour urine calcium

46. Presentation 6) A 28 year old female has recently recovered from her third bout of renal colic. She is otherwise healthy. Investigations reveal no evidence of hyperparathyroidism, urinary tract anomaly or other underlying condition. Which of the following would be most helpful in preventing this from recurring? a) hydrochlorothiazide b) Flomax c) cranberry juice d) 8 glasses of water daily

47. Basic Rules for MCQ’s (A type)* Focus item on important concept Item should assess application of knowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover the options’ test All distractors should be homogeneous Avoid technical item flaws

48. One last step… • Get a colleague to read over your questions • Detects more than one right answer • Discovers the ‘what am I thinking?’ question • Identifies not enough information in the stem • …