160 likes | 444 Views
Are Accommodations Used for ELL Students Valid?. Jamal Abedi University of California, Davis National Center for Research on Evaluation, Standards and Student Testing (CRESST). Validity of accommodations.
E N D
Are Accommodations Used for ELL Students Valid? Jamal Abedi University of California, Davis National Center for Research on Evaluation, Standards and Student Testing (CRESST)
Validity of accommodations • Results of several studies have indicated that some accommodation strategies help both ELL and non-ELL students • These accommodations do more than just “level the playing field” because they provide unfair advantages to the recipients • For some accommodations there is enough research to judge the appropriateness and validity of the accommodations for ELL students
Validity of Accommodations • Research has shown that providing an English dictionary (Abedi, Courtney, & Leon, 2003; Abedi, Lord, Kim, & Miyoshi, 2000) and extra time (Abedi, Lord, Hofstetter, & Baker, 2000; Hafner, 2001; Thurlow, 2001) affects performance of all students (see also, Maihoff, 2002; Thurlow & Liu, 2001) • That is, the results of accommodated and non-accommodated assessment may not be aggregated • Research also suggests that translation of assessment tools into students’ native language may not produce desirable results if the language of instruction and assessment is not aligned (Abedi, Lord, Hofstetter, & Baker, 2000) • The purpose of accommodation for ELL students is to reduce the impact of language factors in order to provide a fair assessment of the content knowledge for these students
This particular study addresses the following four questions: • 1. Do accommodation strategies help reduce the performance gap between ELL and non-ELL students by removing language barriers? (Effectiveness) • 2. Do accommodation strategies impact performance of non-ELL students on content-based assessment? (Validity) • 3. Do student background variables impact performance on the accommodated assessments? (Differential impact) • 4. Are the accommodations easy to apply? (Feasibility)
Methodology • A total of 1,854 Grade 4 students and 1,594 Grade 8 students in 132 classes at 40 school sites participated in this study • The English proficiency designation of students was determined based on school records • Out of 3,448 students, 1,712 (49.7%) students were identified as being ELL • Three language categories were targeted for this study: Spanish, Chinese, and “other Asian languages”
Methodology (continues…) • Of the 1,712 ELL students, 1,614 (94.3%) students had Spanish language background • A science test of multiple-choice and open-ended questions was administered in four forms: (1) original, (2) either a Customized English Dictionary, (3) a Bilingual/English Glossary, or (4) a Linguistic Modification of the test that addressed the challenge of understanding the English lexicon and, possibly, its syntax • These three forms also allowed additional time (50%) • In addition to the science test, an English language reading battery was also administered to determine students’ reading proficiency levels
Methodology (continues…) • The linguistic complexity of the science items—but not the science content—was simplified • The reading measure was important because students at different levels of reading proficiency may benefit differently from any accommodation • The study also included a student background questionnaire, an accommodation follow-up questionnaire, and teacher and school questionnaires • The questionnaire included language background, country of origin, and length of time in the United States, and also asked students to self-assess their proficiency both in English and in their home language
Methodology(continues…) • The accommodation follow-up questionnaire asked students whether the dictionary/glossary helped them during the science test and how the language in the test could have been made easier to understand • To control for teacher, school, and class effects, test materials and accommodations were distributed randomly among students
Null Hypotheses • H01: In the science assessment, ELL students do not benefit from any of the accommodations used in this study. (Effectiveness) • H02: Accommodation does not impact performance of non-ELL students on the science test. (Validity) • H03: Student background variables do not impact performance on accommodated science assessment. (Differential impact)
Methodology (continues…) • We created a latent composite of all the three components that measured English reading efficiency/proficiency: (1) Fluency section of the Language Assessment Scales (LAS), (2) one intact block of the 1994 NAEP Reading assessment (released items), and (3) an experimental word recognition section. • We used this latent composite as a covariate • Each open-ended science and reading item was scored independently by two raters • Interrater reliability indices (percent of exact and within one-point agreement, P.M. correlation, intraclass correlation, and kappa, and alpha coefficients) were computed • For the open-ended science items in Grade 4, percent of agreement ranged from a low of 69% to a high of 97%. The kappa coefficient ranged from a low of .42 to a high of .94
Findings • The internal consistency coefficient for the entire set of Grade 8 reading items was .78; for Grade 4, the overall internal consistency coefficient for reading was .82 • Students with Spanish as a primary home language performed slightly higher on the Linguistic Modification accommodation than under the Standard condition. • Students whose primary home language is neither English nor Spanish benefited the most from the Linguistic Modification version of the test
Item-level analysis • The higher the level of the linguistic complexity, the larger the performance difference between ELL and non-ELL students • The larger the performance difference between ELL and non-ELL students, the more linguistic modification of test items helped reduce the performance gap • Out of 30 science items, students performed better on 22 items under the Linguistic Modification condition
Findings (continue…) • Neither ELL nor non-ELL Grade 4 students benefited from any of the three accommodation strategies that were used • We believe this may be because language demand in textbooks and tests may be less in lower than in higher grades • The lack of significant impact on Grade 4 non-ELL students is an encouraging result because it suggests that the accommodation did not alter the construct under measurement
Results(continues…) • The Linguistic Modification accommodation was shown to have a significant impact on the ELL students’ performance in grade 8 • Accommodation helped ELL students to increase their performance while the accommodated performance of non-ELL students was unchanged
Summary & Conclusion • A non-significant impact of the linguistically modified test on the non-ELL group assures the validity of this accommodation • Increase performance of ELL students (grade 8) under the linguistically modified test shows effectiveness of this accommodation • As for feasibility, this accommodation requires up-front preparation, but is easy to implement in the field; therefore, it is feasible for large-scale assessments
For more information contact: Jamal Abedi: jabedi@ucdavis.edu (530) 754-9150