Are Accommodations Used for ELL Students Valid?

1. Are Accommodations Used for ELL Students Valid? Jamal Abedi University of California, Davis National Center for Research on Evaluation, Standards and Student Testing (CRESST)

2. Validity of accommodations • Results of several studies have indicated that some accommodation strategies help both ELL and non-ELL students • These accommodations do more than just “level the playing field” because they provide unfair advantages to the recipients • For some accommodations there is enough research to judge the appropriateness and validity of the accommodations for ELL students

3. Validity of Accommodations • Research has shown that providing an English dictionary (Abedi, Courtney, & Leon, 2003; Abedi, Lord, Kim, & Miyoshi, 2000) and extra time (Abedi, Lord, Hofstetter, & Baker, 2000; Hafner, 2001; Thurlow, 2001) affects performance of all students (see also, Maihoff, 2002; Thurlow & Liu, 2001) • That is, the results of accommodated and non-accommodated assessment may not be aggregated • Research also suggests that translation of assessment tools into students’ native language may not produce desirable results if the language of instruction and assessment is not aligned (Abedi, Lord, Hofstetter, & Baker, 2000) • The purpose of accommodation for ELL students is to reduce the impact of language factors in order to provide a fair assessment of the content knowledge for these students

4. This particular study addresses the following four questions: • 1. Do accommodation strategies help reduce the performance gap between ELL and non-ELL students by removing language barriers? (Effectiveness) • 2. Do accommodation strategies impact performance of non-ELL students on content-based assessment? (Validity) • 3. Do student background variables impact performance on the accommodated assessments? (Differential impact) • 4. Are the accommodations easy to apply? (Feasibility)

5. Methodology • A total of 1,854 Grade 4 students and 1,594 Grade 8 students in 132 classes at 40 school sites participated in this study • The English proficiency designation of students was determined based on school records • Out of 3,448 students, 1,712 (49.7%) students were identified as being ELL • Three language categories were targeted for this study: Spanish, Chinese, and “other Asian languages”

6. Methodology (continues…) • Of the 1,712 ELL students, 1,614 (94.3%) students had Spanish language background • A science test of multiple-choice and open-ended questions was administered in four forms: (1) original, (2) either a Customized English Dictionary, (3) a Bilingual/English Glossary, or (4) a Linguistic Modification of the test that addressed the challenge of understanding the English lexicon and, possibly, its syntax • These three forms also allowed additional time (50%) • In addition to the science test, an English language reading battery was also administered to determine students’ reading proficiency levels

7. Methodology (continues…) • The linguistic complexity of the science items—but not the science content—was simplified • The reading measure was important because students at different levels of reading proficiency may benefit differently from any accommodation • The study also included a student background questionnaire, an accommodation follow-up questionnaire, and teacher and school questionnaires • The questionnaire included language background, country of origin, and length of time in the United States, and also asked students to self-assess their proficiency both in English and in their home language

8. Methodology(continues…) • The accommodation follow-up questionnaire asked students whether the dictionary/glossary helped them during the science test and how the language in the test could have been made easier to understand • To control for teacher, school, and class effects, test materials and accommodations were distributed randomly among students

9. Null Hypotheses • H01: In the science assessment, ELL students do not benefit from any of the accommodations used in this study. (Effectiveness) • H02: Accommodation does not impact performance of non-ELL students on the science test. (Validity) • H03: Student background variables do not impact performance on accommodated science assessment. (Differential impact)

10. Methodology (continues…) • We created a latent composite of all the three components that measured English reading efficiency/proficiency: (1) Fluency section of the Language Assessment Scales (LAS), (2) one intact block of the 1994 NAEP Reading assessment (released items), and (3) an experimental word recognition section. • We used this latent composite as a covariate • Each open-ended science and reading item was scored independently by two raters • Interrater reliability indices (percent of exact and within one-point agreement, P.M. correlation, intraclass correlation, and kappa, and alpha coefficients) were computed • For the open-ended science items in Grade 4, percent of agreement ranged from a low of 69% to a high of 97%. The kappa coefficient ranged from a low of .42 to a high of .94

11. Findings • The internal consistency coefficient for the entire set of Grade 8 reading items was .78; for Grade 4, the overall internal consistency coefficient for reading was .82 • Students with Spanish as a primary home language performed slightly higher on the Linguistic Modification accommodation than under the Standard condition. • Students whose primary home language is neither English nor Spanish benefited the most from the Linguistic Modification version of the test

12. Item-level analysis • The higher the level of the linguistic complexity, the larger the performance difference between ELL and non-ELL students • The larger the performance difference between ELL and non-ELL students, the more linguistic modification of test items helped reduce the performance gap • Out of 30 science items, students performed better on 22 items under the Linguistic Modification condition

13. Findings (continue…) • Neither ELL nor non-ELL Grade 4 students benefited from any of the three accommodation strategies that were used • We believe this may be because language demand in textbooks and tests may be less in lower than in higher grades • The lack of significant impact on Grade 4 non-ELL students is an encouraging result because it suggests that the accommodation did not alter the construct under measurement

14. Results(continues…) • The Linguistic Modification accommodation was shown to have a significant impact on the ELL students’ performance in grade 8 • Accommodation helped ELL students to increase their performance while the accommodated performance of non-ELL students was unchanged

15. Summary & Conclusion • A non-significant impact of the linguistically modified test on the non-ELL group assures the validity of this accommodation • Increase performance of ELL students (grade 8) under the linguistically modified test shows effectiveness of this accommodation • As for feasibility, this accommodation requires up-front preparation, but is easy to implement in the field; therefore, it is feasible for large-scale assessments

16. For more information contact: Jamal Abedi: jabedi@ucdavis.edu (530) 754-9150