Standards for Question Writing

1. Standards for Question Writing Mary Grantner, MA Senior Manager, Physician Assessment

2. Fundamentals What is a test? An instrument that tells us something about student knowledge, skills, and/or abilities.

3. Fundamentals What is a test item? Cue requiring student response with rules for scoring response. Response can be prompted by an instruction to perform a task, or a question requiring an answer.

4. Fundamentals By definition, ARS employs a question requiring an answer. The default test item is multiple-choice. • Stem • Responses • Single correct response (best-answer format)

5. Use those nifty keypads A multiple-choice test item can measure which of the following student achievements? • Critical thinking skill • High-inference psychomotor skill • Declarative knowledge • High-inference cognitive ability

6. A multiple-choice test item can measure which of the following student achievements? • Critical thinking skill* • High-inference psychomotor skill • Declarative knowledge* • High-inference cognitive ability*

7. Approach to Writing • Identify Key Concepts • What do you want to measure? • Identify Actions • What do you want the learner to be able to do? • Ask for Direct Action/Concept • Provide a Single Correct Answer

8. Identify Key Concepts • Start with learning objectives • When they walk out of the room, will learners ask “so what?” What should they now understand? • Put each concept into a declarative sentence.

9. Identify Actions • Drill down – what should the learner do with the concept? • Verbs “do” – consider some verbs that apply to the Key Concept. • Put each action into a declarative sentence.

10. A Word about Verbs • Strong verbs are vital to any good writing. • Be direct and active. • Form of “to be” + past participle – Mary was given a book by John – is passive. • Avoid forms of “to be” when possible. • And (drumroll please)…

11. “When you see an adverb, kill it.” -- Mark Twain What does it mean to walk quickly? Briskly? Urgently? Fast?

12. Choosing Verbs Measure Recall or Understanding, use: Characterize Define Demonstrate Describe Find Illustrate

13. Choosing Verbs Measure Problem-solving, use: Classify Conclude Diagnose Determine Plan Recommend

14. Ask for Direct Action/Concept • Focus on a single Key Concept at a time. • Less really is more. • Using an active verb, ask a direct question. • Ask the learner to do something. • Keep it simple!

15. Provide a Single Correct Answer • Focus on “real life”. • Be as brief as possible. • Answer should contain only what is absolutely necessary. • Keep it simple!

16. Keypads Up! Which of the following best defines adverb? • Modifies a noun or other substantive. • Ob to be executed by item writer. • Expresses action, existence, or occurrence. • Modifies a verb, adjective, or other adverb.

17. Which of the following best defines adverb? • Modifies a noun or other substantive. • Obliged to be executed by item writer. • Expresses action, existence, or occurrence. • Modifies a verb, adjective, or other adverb. *

18. Try This Identify Key Concept: Pulmonary embolism (PE) and deep venous thrombosis (DVT) are part of the same complex and common process (i.e. venous thromboembolism), which has a wide spectrum of clinical presentations and consequences, ranging from no symptoms or sequelae to a lethal outcome. DVT usually starts in the deep calf veins and propagates superiorly, and it is associated with symptomatic PE in about one-third of untreated patients. DVT is commonly confused with other conditions such as cellulitis, leg edema, and chronic venous insufficiency, and PE is also notoriously difficult to diagnose.

19. You Decide • As the content expert, it’s up to you to decide what is most important. • Let’s choose this: PE is also notoriously difficult to diagnose • Write a direct question based on the paragraph.

20. The Question Which of the following statements best explains the difficulties in diagnosing lower extremity DVT? Answer: Is clinically similar to other conditions of the lower extremities.

21. Wrong Answers • Also called distractors, are hard to write. • Try using these: • Common misconceptions you’ve seen; • Misapply a principle (i.e. add when you should subtract); • Ways in which things are related (cause or coincidence?).

22. Which of the following statements best explains the difficulties in diagnosing lower extremity DVT? Is a distinctly different process than PE. Occurs rarely in cases of symptomatic PE. Is clinically similar to other conditions of the lower extremities.* Usually originates in difficult to image deep veins of the thigh.

23. Other Rules/Hints • Just as in life, a fact is seldom absolutely true or absolutely false. No T/F questions! • “All / None of the Above” is essentially T/F – Not allowed. • Avoid any negative question • Which is false? • Which is incorrect?

24. Other Rules/Hints • Stem is short and options long – a poor item! • As much information as possible in the stem and keep the options short • Avoid absolute terms. • Words like “always” or “never” are rarely true in life and so are almost never the right answer on a test. • Avoid word repeats (word that appears in both stem and response); points to a correct answer. • Language in the options must be parallel (i.e. all diagnoses, or all anatomic entities).

25. Questions? Mary Grantner mgrantner@rsna.org 630-368-3733