Michael C. Rodriguez University of Minnesota CCSSO’s National Conference on Student Assessment

1. Michael C. Rodriguez University of Minnesota CCSSO’s National Conference on Student Assessment June 2008 Modified Achievement Testsfor Students with Disabilities:Distractor Analysis

2. Item Writing: Assessment Building Blocks • Item writing is an art and a science • Item writing requires supervised training • Empirical research on item writing began in 1920s • Multiple-choice items are capable of measuring a wide range of content and cognitive domains in a short time with a high level of reliability • Not all students are able to perform on today’s multiple-choice tests without accommodations or modifications

3. Item Modification • Much of the language surrounding test item modification suggests that the goal is to make items “easier”. • But making items easier doesn’t necessarily improve measurement. • Elements of Universal Design provide a model for making appropriate modifications. • Empirical research on item writing provides an evidence basis for item modifications.

4. Measurement & Access • Test items provide opportunities for students to display construct-relevant knowledge, skills, and abilities. • From a measurement perspective, item modifications should be done to • Provide access to the item for all students • Improve measurement of the construct • The hypothesis is: By providing greater access to each item, we improve measurement.

5. Number of Options • Less time is needed to prepare 2 plausible distractors than 3 or 4 distractors • More 3-option items can be administered within the same time limit than 4 or 5-option items, improving content coverage • Evidence suggests no significant reduction in test item or test score quality by reducing the number of options

6. Item Difficulty

7. Item Discrimination

8. Test Score Reliability

9. Sample Item Pesticides In the late 1980s, farmers began to use a pesticide to control insects that harmed their cotton crops. This problem was solved. However, an insect group that pollinated the corn crops was also injured. Without pollination the corn kernels did not fully develop. This affected the corn harvest on which the farm families had come to depend. What is not mentioned as one effect of pesticide usage?A. soil contamination B. destruction of pests C. destruction of friendly insects D. crop losses

10. Pesticides In the late 1980s, farmers began to use a chemical pesticide. It was used to control insects that harmed their cotton crops. This solved one problem, but caused another. An insect group that pollinated the corn crops was also harmed by the pesticide. Without pollination the corn kernels did not fully develop. This decreased the corn harvest. What is NOT mentioned as one effect of using chemical pesticides? A. destruction of the soilB. destruction of pestsC. destruction of friendly insects

11. CAAVES Reliability Results • All forms retained an acceptable level of reliability – for 39 item tests • .85 for Mathematics • .90 for Reading • Significant Group x Condition Interaction • Test score reliability depended on the Group (Disabled v. Nondisabled students) and Condition (Modified v. Unmodified forms) combination

12. Group x Condition Interaction Disabled/Nondisabled differences • Mathematics • Modified Forms: .025 • Unmodified Forms: .027 • Reading • Modified Forms: .029 • Unmodified Forms: .020

13. Group x Condition Interaction The magnitude of differences in Reliability differed by Group x Condition combination, but appeared to be a function of the “Eligible” group

14. Mathematics

15. Reading

16. Spearman’s Rho Correlations between Change in Location and Difference in Item p-value Correct Option Location

21. Issues related to Access • Fewer options reduce the cognitive load • More options result in exposing additional aspects of the domain to students – possibly providing clues to other questions • More options can introduce irrelevant aspects of the domain

22. Method of Option Deletion • Examining two common deletion methods (distractor functioning and random deletion) • Going from 5 to 3 options: • No relation between deletion method and item difficulty or discrimination (effect homogeneity) • No change in reliability for deletion of ineffective distractors (.006) • Random deletion resulted in a reduction in reliability of .06

23. Improving Diagnostic Information • Distractors that are written to be plausible should contain common errors or misconceptions • Distractor analysis provides information regarding the kinds of errors or misconceptions held by students • No reason, psychometrically, to have the same number of options for every item

24. References Rodriguez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13. Haladyna, T.M., Downing, S.M., & Rodriguez, M.C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-334.