National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007

1. National Accessible Reading Assessment Projects General Advisory Committee December 7, 2007 Overview of DARA Project Research Linda Cook

2. Background • Research Questions • Do students with reading-based learning disabilities receive differential performance gains from an audio (read aloud) accommodation/modification when compared to students without disabilities? • Differential boost • Do reading comprehension tests and test items taken with and without a read aloud accommodation/modification measure similar constructs? • Factor analysis • Differential item functioning analysis • Distractor analysis • Cognitive labs

3. Background • DARA Tests and Samples • Two different tests and several different samples used for the studies • Gates-MacGinitie Reading Test (GMRT) • Fourth and eighth grade students (randomized design) • State standards-based assessment of English-language arts (ELA) • Fourth and eighth grade students (archival data)

4. Summary of DARA Analyses

5. Overview of Presentation • Gates MacGinitie Reading Test (GMRT) • Differential boost • DIF and factor analysis studies • English-language arts assessment (ELA) • DIF and factor analysis studies • Next steps

6. Differential Boost Description • Framework for studying impact of changes in testing conditions (accommodations or modifications to the standard testing procedures) • A change in testing condition may be viewed as an accommodation if students with disabilities receive a significant increase in scores compared to students without disabilities

7. Differential Boost Study • Two forms of GMRT (reading comprehension section only) • Form S • Form T • Two grades • Grade 4 • Grade 8 • All forms and grade levels contained 48 passage based questions • All questions were multiple-choice

8. Differential Boost Samples • All NJ public school 4th and 8th grade students were invited to participate • Samples • Fourth grade • 1170 students • 522 (RLD) • 648 (NLD) • Eighth grade • 855 students • 394 (RLD) • 461 (NLD)

9. Differential Boost Data Collection Design

10. Grade 4 Differential Boost Analysis

11. Grade 8 Differential Boost Analysis

12. Differential Boost ANOVA • Results of RM-ANOVA showed differential boost • Students with disabilities had significantly larger boost than students without disabilities • Significant interaction between score gain (boost) and disability status was observed • Answer to first question: Students with reading based learning disabilities do exhibit differential performance gains when they take a reading test with a read aloud test change

13. Purpose of Differential Item Functioning and Factor Analyses of GMRT • Determine if the test is measuring the same underlying construct (s) for the following comparisons: • Students without disabilities who take the test with and without a read aloud test change • Students with disabilities who take the test with and without a read aloud test change

14. Differential Item Functioning and Factor Analysis of GMRT Data • Comparisons of item responses for four groups • Reading based learning disability (RLD), no accommodation (Standard) • Reading based learning disability (RLD), audio modification (Audio) • No disability (NLD), no accommodation (Standard) • No disability (NLD), audio modification (Audio)

15. Differential Item Functioning of GMRT Data • Analyzed Form S (4th and 8th grade levels) and Form T (4th and 8th grade levels) • Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change • Does the GMRT measure the same underlying construct (s) for the study groups?

16. Summary Statistics for Grade 4 Form S DIF and Factor Analysis Samples

17. Differential Item Functioning (DIF) • Test takers matched on proficiency level • Used total test score as matching criterion • Reference Groups • Students with and without disabilities who took test under standard conditions • Focal Groups • Students with and without disabilities who took test with read aloud test change

18. Differential Item Functioning (DIF) Analyses • Used Mantel-Haenszel procedure with total score as criterion • Mantel-Haenszel categorization • A—negligible DIF • B—slight to moderate DIF • C—moderate to large DIF • Direction of DIF Flags • Negative favors reference group • Positive favors focal group

19. Comparison 1: RLD Standard (reference group) vs. RLD Audio (focal group)

20. Comparison 2: NLD Standard (reference group) vs. NLD Audio (focal group)

21. Factor Analysis of GMRT Data • Analyzed Form S (4th and 8th grade levels) and Form T (4th and 8th grade levels) • Investigated the impact of a read aloud test change for students with and without disabilities who took the test with and without the test change • Does the GMRT measure the same underlying construct (s) for the study groups?

22. Factor Analysis of GMRT Data • Single group exploratory factor analysis of four groups • RLD who took the test under standard conditions • RLD who took the test with an audio test change • NLD who took the test under standard conditions • NLD who took the test with an audio test change • Confirmatory factor analysis • Single-group to establish single factor • Multi-group to establish base line model • Multi-group comparisons to establish measurement invariance • Comparison 1—RLD standard vs. RLD read aloud • Comparison 2—NLD standard vs. NLD read aloud

23. Summary of Multi-group CFA for RLD Standard vs. RLD Audio

24. Summary of GMRT Analyses • Purpose of the studies was to determine if the test measures the same underlying construct (s) when taken with and without a read aloud test change • Results of the differential boost study showed that students with reading based learning disabilities receive a differential boost on a reading test taken with a read aloud test change • Results of DIF and factor analysis studies indicate that test measures the same underlying construct for the RLD group, or for the NLD group when taken with and without a read aloud test change

25. Analyses of English-language Arts Assessment • Purpose • To examine if the test measures the same construct (s) for • Students without disabilities • Students with learning disabilities who took the test with and without an accommodation/modification

26. Analyses of ELA Assessment • Analyses • Factor analyses • Differential item functioning analyses and distractor analyses • Groups of Interest • Students with learning disabilities who took the test with and without a change in testing conditions • Test • Grade 4 and grade 8 English-language Arts (ELA) assessment • Focus • Determine if the test measures the same constructs for • Examinees without disabilities • Examinees with learning disabilities who took the test with and without a change in testing conditions

28. English-Language Arts Grade 4 Summary Statistics

29. Factor Analyses of ELA Assessment Purpose: to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study • Exploratory analyses (separately in each group) • how many factors • Confirmatory (multi-group) • Establish base-line model • Confirm number of factors needed to describe data across all groups

30. Differential Item Functioning (DIF) Analyses • The purpose of the DIF study was to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study • Used Mantel-Haenszel procedure with total score as criterion • Mantel-Haenszel categorization • A—negligible DIF • B—slight to moderate DIF • C—moderate to large DIF • Direction of DIF Flags • Negative favors reference group • Positive favors focal group

31. Comparisons Made in the DIF Study

32. Interpreting the Results of the DIF Study • Fourth grade results • 1 C DIF item, 10 B DIF items • 5 B DIF items were reading items that favored students with disabilities who took test with read-aloud change in testing conditions • 5 B DIF items (3 reading and 2 writing) favored students without disabilities • Results consistent with factor analyses results

33. Interpreting the Results of the DIF Study • Eighth grade results • 1 C DIF item, 7 B DIF items • Five B DIF items (4 reading and 1 writing) favored students who took test with read-aloud change in testing conditions • Again, results consistent with factor analyses results

34. Summary of Research Results • GMRT • Differential boost study • Students with reading disabilities show differential performance gains when taking a test with read aloud • DIF and Factor Analysis • Internal structure (constructs) measured by test taken by students with and without disabilities taking test with and without read aloud very similar • ELA • DIF and Factor Analysis • Internal structure (constructs) measured by test taken by students with and without disabilities taking test with and without test changes very similar

35. Research Answers • Do students with reading-based learning disabilities receive differential performance gains from read aloud? • yes • Do reading tests and test items taken with and without a read aloud accommodation/modification measure similar constructs? • Measure the same single factor for all groups • Show little evidence of differential item performance

36. Next Steps • Complete analysis of data from cognitive labs • Design and carry out simulations for multi-stage reading assessment • Design and carry out field trial of prototype assessment

37. Analyzing Data from Cognitive Labs • Cognitive labs are a means of measuring mental processes through a think aloud protocol • Administered GMRT to 34 fourth grade students and 15 eighth grade students • Gained insight into test taking strategies as well as issues related to test format and content

39. NARAP Field Trial • Large data collection effort • Design and develop accessible reading assessment prototypes based on NARAP Principles and Guidelines • Will field trial DARA, PARA, and TARA prototypes

40. Questions?