Item Analyses and Features Studies and Other Findings

1. Item Analyses and Features Studies and Other Findings October 13, 2006 Jamal Abedi CRESST/University of California, Davis Seth Leon & Jenny Kao CRESST/University of California, Los Angeles

2. Examining Differential Item Functioning in Reading Assessments for Students with Disabilities • This study examined differences between students with disabilities and non-disabled students using DIF • Data from Stanford Achievement Test (Ninth Edition) Reading Comprehension (RC) and Word Analysis (WA) in two different states were used • Data were from the 1997-1998 school year

3. Focus on Reading • In Site 2 there were 278,287 Grade 3 students (7.6% with disabilities) and 244,446 Grade 9 students (9.9% with disabilities). • In Site 4, there were 6,611 Grade 3 students (6.8% with disabilities), and 5,287 Grade 9 students (9.9% with disabilities)

4. The following research questions guided the study: • Do items on standardized Reading Comprehension (RC) and Word Analysis (WA) subscales exhibit Differential Item Functioning (DIF) for students with disabilities? • Are more items that exhibit DIF for students with disabilities located in the second half of RC and WA subscales rather than in the first half?

5. Research questions (continued) 3. Do students with disabilities consistently under-perform on items located in the second half relative to items located in the first half, as compared to non-disabled students? 4. Do the results of DIF vary by grade (3 and 9)?

6. Results • For grade 9, many items exhibited DIF • Items that exhibited DIF were more likely to be located in the second half of the assessment subscales • After controlling for reading ability, students with disabilities consistently under-performed on items located in the second half • Results were seen in grade 9 for data from two different states.

7. Results (continued) • In grade 3 there were fewer items that were shown to exhibit DIF for students with disabilities This was true for both RC and WA subscales • This study has several limitations to the data: • There was no access to information regarding the testing accommodations that students with disabilities might have received • No access to the type of disabilities

8. Examining Differential Distractor Functioning in Reading Assessments for Students with Disabilities • In DDF analysis we examined the pattern of incorrect answers or distractors • Data from Site 4 were used to examine Grade 3 and Grade 9 students’ responses to Stanford Achievement Test, Ninth Edition.

9. The following research questions guided this study • Do items on standardized Reading Comprehension (RC) and Word Analysis (WA) subscales exhibit differential distractor functioning (DDF) for students with disabilities? • Does the differential distractor functioning for students with disabilities increase for items located in the second half of RC and WA subscales? • Do the results of DDF vary by grade (from grade 3 to grade 9)?

10. Results • Results suggest that a substantial number of items from both the Reading Comprehension (RC) and Word Analysis (WC) subscales exhibit DDF for students with disabilities in grade 9. • Results also suggest that items showing DDF were more likely to be located in the second half of the assessments rather than the first half of the assessments. • Results also indicate that DDF was present for grade 9 test items, but not for grade 3 items.

11. Results (continued) • Even when controlling for ability using only the items in the first half of the assessments, more grade 9 items exhibited DDF than grade 3 items • Students with disabilities were less likely to choose the most common distractor chosen by their non-disabled peers • Students with disabilities might be more randomly selecting one of the four response options rather than making an “educated guess”

12. Results (continued) • Access to information regarding the type of accommodations students received (e.g.,extended time) would be helpful, but was not possible

13. Summary/Conclusion • More items were identified as DIF and DDF in grade 9 than grade 3 • Students with disabilities selected less commonly used distractors • In both DIF and DDF students with disabilities had more difficulties with the items in the second half of the test • This may suggest the higher level of impact of fatigue and frustration on students with disabilities

14. Item Features Study Research Questions: Focus on Passage Order (Model #1) • Test versions: two for students with disabilities (1-2; 2-1); three for non-disabled students (1-2 and 2-1 and 1 chunked-2 not-chunked). • The research questions are divided into three parts even though some of them may be inter-related.

15. I.Passage Order (1-2 versus 2-1) • Does passage order affect students’ performance overall? • Is student performance on the first passage affected by whether it appears first in the test (1-2) or last (2-1)? • Is there a differential effect of passage order on students with disabilities as compared with non-disabled students?

16. II. Motivation & Fatigue • Do the students with disabilities in this study report lower motivation than non-disabled students? • Does the performance of students who reported lower motivation have a relationship with passage order? • Is there a relationship between passage order and students’ rating of fatigue?

17. III. Chunking (for non-disabled students; passages 1-2 both chunked) • Is student performance affected by whether the test is presented as chunked or not chunked? • Is there a relationship between students’ report of motivation and chunking? • Is there a relationship between students’ report of fatigue and chunking? Note: Fatigue questions will appear after each set of passages and corresponding items. Motivation questions appears once as post-scale.

18. Design • Analysis of covariance • Student ability will be controlled for using scores on their prior year’s state assessment, or by a quick reading efficiency battery (the CRESST TIMER test) • Motivation scale will appear only as a post-test. • Research questions relating to chunking will again be addressed with a similar analysis of covariance approach. Analysis will only be applied to the students without disabilities.

19. Item Features Study Research Questions: Focus on Chunking (Model #2) • Will be tested on students with no disabilities • Test versions: two for everyone; both consist of two long passages; one version is chunked, one is not chunked.

20. I. Chunking • Does chunking improve the performance of non-disabled students? (validity) • Does chunking improve the performance of students with disabilities? • If the performance of both groups improve, is there a differential effect for students with disabilities?

21. II. Motivation & Fatigue • Do the students with disabilities in this study report lower motivation than non-disabled students? • Did the students who took the chunked version of the test report higher motivation than students who took the non-chunked version?

22. Motivation & Fatigue (continued) • Do the students with disabilities in this study report less fatigue than non-disabled students? • Did the students who took the chunked version of the test report less fatigue than students who took the non-chunked version?

23. III. Passage Order • Is there a difference in performance between students with disabilities and non-disabled students on the second passage after controlling for performance on the first passage? • If so, does this difference in performance occur on both the chunked and non-chunked versions?

24. Limitations • No interaction between passage order and chunking can be estimated • The effect of chunking will only be examined on non-SD students to provide some validity evidence • Small number of SDs threaten to power of the analyses

25. Examine Chunking on Students with Disabilities (Model #3) • Test SDs twice • Test passage order and chunking in a counter-balance order • This model allows examination of chunking on SDs as well as non-SDs • Useful information can be provided, for example, chunking can be more effective for SDs

26. Your advice/suggestions • How to resolve the sample size with the limited access to schools? • Which model do you think would produce more useful results? • Can you help with more school sites/access? • Are you interested in a more involved role in this study?