Authoring High Quality Digital Multiple-Choice Assessments in Online Courses

1. Authoring High Quality Digital Multiple-Choice Assessments in Online Courses Susan M. Moncada, Indiana State University Thomas P. Moncada, Eastern Illinois University MERLOT International Conference August 9, 2008

2. Test Item Design Jeopardy Stem Flaws Stats Analysis Distractor Avoidance Clueing Concerns 100 100 100 100 200 200 200 200 300 300 300 300 Adapted from MERLOT artifact 78866, Financial Accoun- ting Jeopardy Games, by Mike Seda, Shaw University

3. Stem Flaws 100 The answer is…Which of the following is not a MERLOT review criteria? Q: What is a negatively worded stem? • Click once to display the question • Click on the arrow to return to the game

4. Stem Flaws 200 The answer is…. MERLOT is a. A line of skin care products b. A producer of customized tarping systems c. A repository of digital learning objects d. A term for “black bird” in French Q: What is an unfocussed stem? • Click once to display the question • Click on the arrow to return to the game

5. Stem Flaws 300 The answer is….unnecessary irrelevant content in a multiple-choice stem. Q: What is window dressing? • Click once to display the question • Click on the arrow to return to the game

6. Distractor Avoidance 100 • The answer is… • MERLOT membership benefits include • Personal profiles and blogs • Personal collections and personal profiles • Second Life access and personal collections • Virtual speakers’ bureau and free software Q: What are overlapping distractors? • Click once to display the question • Click on the arrow to return to the game

7. Distractor Avoidance 200 • The answer is… • Who is the Executive Director of MERLOT? • Carl Sandburg • Gerry Hanley • Abraham Lincoln • Spiderman Q: What are implausible distractors? • Click once to display the question • Click on the arrow to return to the game

8. Distractor Avoidance 300 The answer is…always, never, totally, absolutely, and completely. Q: What are specific determiners? • Click once to display the question • Click on the arrow to return to the game

9. Clueing 100 • The answer is… • Which criteria does “Ease of Use” include? • Concepts are completely demonstrated • Content is well summarized • Having clear instructions • Prerequisite is knowledge identified Q. What are grammatical inconsistencies? • Click once to display the question • Click on the arrow to return to the game

10. Clueing 200 • The answer is… • Which reviewer phrase is a teaching effectiveness strength? • Requires context • Uses post-graduate vocabulary • Demonstrates relationships between all concepts • Provides interactivity Q: What is a longest response key ? • Click once to display the question • Click on the arrow to return to the game

11. Clueing 300 • The answer is… • Which criteria does “Quality of Content” include? • Accurate content • Visual appeal • Potential effectiveness • Prerequisite knowledge Q: What is a clang association? • Click once to display the question • Click on the arrow to return to the game

12. Stats Analysis 100 The answer is… A Kuder-Richardson (KR-20) score >= .70. Q:What is good internal consistency for teacher made tests? • Click once to display the question • Click on the arrow to return to the game

13. Stats Analysis 200 The answer is… An IDR of .6 and a P-value of .5. Q: What is a multiple choice question that discriminates well and is of average difficulty? • Click once to display the question • Click on the arrow to return to the game

14. Stats Analysis 300 The answer is…1 out of 285 Q: What is the chance of scoring > 70% on a ten item multiple choice assessment with 4 distractors? • Click once to display the question • Click on the arrow to return to the game