Mary Pitoniak, Linda Cook, Frederic Cline, and Cara Cahalan-Laitusis

1. Using Differential Item Functioning to Investigate the Impact of Accommodations on the Scores of Students with Disabilities on English-Language Arts Assessments Mary Pitoniak, Linda Cook, Frederic Cline, and Cara Cahalan-Laitusis Educational Testing Service NCME Presentation April 10, 2006

2. Purpose and Overview of the Study • The purpose of this study was to examine differential item functioning on the English-Language Arts assessment described by Linda • DIF analyses are statistical procedures that are used to identify items that function differently for different subgroups of examinees • DIF “exists when examinees of equal ability differ, on average, according to their group membership in their responses to a particular item” (Standards)

3. Purpose and Overview of the Study (continued) • Issues investigated: • Do 2 different DIF detection methods yield the same results? • Are the results interpretable in terms of a priori or a posteriori evaluation of item content? • Of particular interest:When the read-aloud modification is used, do the items function differentially for students?

4. Purpose and Overview of the Study (continued) • Features of study: • 2 DIF detection methods • Large enough sample sizes (not always the case) • Looked at 3 different criteria (total score, Reading score, Writing score); we decided to go with total score for several reasons • Used purification step, as recommended by literature

5. Comparisons Made in the Study

6. DIF Methods Used • Mantel-Haenszel • Logistic Regression • For both methods, we used ETS classification system: • Category A contains items with negligible DIF; • Category B contains items with slight to moderate values of DIF; • Category C contains items with moderate to large values of DIF.

7. Comparison of Mantel-Haenszel vs. Logistic Regression

8. Example of Uniform DIF

9. Example of Non-Uniform DIF

10. Results • Within this presentation, I will present results only for Reading items (and not Writing), both for time reasons and because we were most interested in the effects of the accommodations on performance on the Reading items

11. Results (continued) • Overall • No items flagged as “C” • Each method flagged 9 items as “B” (out of 42 items X 5 comparisons, or 210 possible flags) • However, those 9 items were not the same items—in all, 12 different items were flagged by at least one of the methods • There were inconsistencies between methods

12. Number of Items Flagged by Each Method

13. Agreement Between Flags for Methods by Comparison Type

14. Non-LD vs.LD No Accommodation

15. Non-LD vs.LD IEP/504 Accommodation

16. Non-LD vs.LD Read-Aloud Modification

17. LD Non-Accommodatedvs.LD IEP/504 Accommodation

18. LD Non-Accommodatedvs.LD Read-Aloud Modification

19. Example of Discrepancies in Flags Item Flags M-H—Uniform LR—No flag The items flaggedby MH (but not LR) as favoring students with read-aloud modification did show differences such as these graphically for LR

20. Example of Discrepancies in Flags Item Flags M-H—Uniform LR—No flag

21. A Priori Theories About Read-Aloud Modification Results • 5 items were easier for students who received the read-aloud modification than for non-LD students. • A priori theories were not that accurate! • Item A: harder (refer back) • Item B: easier (short item; intonation/body language) • Item C: easier (intonation/body language) • Item D: harder (char. of options) • Item E: harder (length of options)

22. A Posteriori Interpretation About Read-Aloud Modification Results • The reasons why these 5 items were easier with read-aloud accommodation were not obvious to test developers

23. What Do the Results Say About the 3 Questions Posed • Do 2 different DIF detection methods yield the same results? • Neither flagged an item as “C.” • There were discrepancies in “B” flags, however. • Some discrepancies are explainable in terms of advantages/disadvantages of methods as listed earlier.

24. 3 Questions (continued) • Are the results interpretable in terms of a priori or a posteriori evaluation of item content? • Not consistently • Of particular interest:When the read-aloud modification is used, do the items function differentially for students? • Yes, some items were easier when read-aloud, which supports this state’s decision to view read-aloud as a modification

25. Next Steps • ELL and ELL/LD groups to be compared • Grade 8 ELA to be evaluated • DIF analysis paradigm to be utilized