ABQAURP Item Writing Workshop

1. Presented by Scott E. Arbet PhD ABQAURP Item Writing Workshop

2. Stem (Vignette + Lead-in Question) Spurred by the Institute of Medicine reports on medical errors, a hospital's governing body demands a crash program to reduce medical errors. The chief executive officer orders the senior staff to immediately start nine patient safety projects. This decision will impede the likelihood of the organization lowering the rate of medical errors for which of the following reasons? Anatomy of an Item

3. A-type Items • Vignette • Components • Patient age and gender if applicable • Site of care or facility (office, hospital, ED, etc . . . . ) • Presenting symptoms, issues, case history • Pertinent history such as medical care, fraud, consent forms, etc. • Specific law or case-related information

4. A-type Items • Vignette • Rules • Focus on a single, clearly defined topic. • Exclude irrelevant information (avoid “window dressing” and “red herrings”). • Do not use abbreviations, jargon, or unnecessary complexity.

5. A-type Items • Lead-in Question • Rules • Ask a focused question. • Use superlatives such as most or best. • Use positively worded lead-ins. • Do not write true-false lead-ins such as: Which of the following statements about X is true? • Do not use may and may happen. • Do not use indefinite words such as frequently and often. • Do not use absolutes such as never, none, no, all, always, and every. • Do not ask which of the following is true EXCEPT (note to update web site practice questions).

6. A-type Items • Lead-in Question • Cover the Options Rule: You know you have written an effective and focused lead-in question if it is possible to arrive at an answer without looking at the options.

7. A-type Items • Examples of Focused Lead-in Questions • Which of the following is the most appropriate management? • Under which of the following conditions are physicians allowed to disclose confidential patient information? • Which of the following is the most important additional question to ask the patient? • Which of the following variables is used to determine which Diagnosis Related Group (DRG) is associated with an inpatient admission?

8. A-type Items • Options • Choose one option as the correct answer. • Select three to four other options (distractors) that are plausible, but incorrect. The true mark of a focused and well-written item is when all of the options have been chosen by some examinees.

9. Application of Knowledge vs Recall • Disadvantages of Recall Items • Not as indicative of a real-life meaningful situation. • Only test memory of facts and not the knowledge or application of those facts. • Do not typically test well. In other words, they do not separate the high scorers from the low scorers.

10. Item Writing/Review Process

11. Item Writing/Review Process • Train writers • Analyze current pool and make assignments • Write 2-3 items • Edit items and provide feedback to authors • Write items

12. Item Writing/Review Process • Blind review • Collect comments, analyze results, and make final adjustments via conference call • Administer items • Key validation of outliers (should be minimal if any due to blind review) • Score

13. Questions?Thank you!