Book 3: Use of Accommodations in Large-Scale Assessments

1. Book 3: Use of Accommodations in Large-Scale Assessments Practical Guidelines for the Education ofEnglish Language Learners

2. The Center on Instruction is operated by RMC Research Corporation in partnership with the Florida Center for Reading Research at Florida StateUniversity; Horizon Research, Inc.; RG Research Group; the Texas Institute for Measurement,Evaluation, and Statistics at the University of Houston; and the VaughnGross Center for Reading and Language Arts at the University of Texas at Austin.The contents of this PowerPoint were developed under cooperative agreement S283B050034 withthe U.S. Department of Education. However, these contents do not necessarilyrepresent the policy of the Department of Education, and you should notassume endorsement by the Federal Government.2007The Center on Instruction requests that no changes be made to the content or appearance of this product.To download a copy of this document, visit www.centeroninstruction.org

3. Practical Guidelines for the Education of English Language Learners Authors David J. Francis, Mabel Rivera Center on Instruction English Language Learners Strand Texas Institute for Measurement, Evaluation, and Statistics University of Houston Nonie K. Lesaux, Michael J. Kieffer Graduate School of Education Harvard University Héctor Rivera Center on Instruction English Language Learners Strand Texas Institute for Measurement, Evaluation, and Statistics University of Houston

4. Practical Guidelines for the Education of English Language Learners Research-based Recommendations for Instruction and Academic Interventions Research-based Recommendations for Serving Adolescent Newcomers Research-based Recommendations for the Use of Accommodations in Large-scale Assessments

5. Book 3: Use of Accommodations in Large-Scale Assessments Foreword Overview Accommodations and Review of State Policies Conceptual Framework Use of Accommodations Meta-Analysis on Effectiveness of Accommodations in Assessment Selected studies for analysis Methods Results Conclusions Technical Appendices

6. Seminal Research Reviews Rivera, C., Collum, E., Shafner Willner, L., & Sia, J. K. (2006). An analysis of state assessment policies regarding the accommodation of English language learners. In C. Rivera and E. Collum (Eds.), State assessment policy and practice for English language learners: A national perspective (pp. 1-173). Mahwah, NJ: Lawrence Erlbaum. Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1-28. Sireci, S., Li, S., & Scarpati, S. (2003). The effect of test accommodation on test performance: A review of the literature (Research Report No. 495). Amherst, MA: University of Massachusetts School of Education, Center for Educational Assessment.

7. Demographics

8. Who areEnglish Language Learners? • National-origin-minority students with limited proficiency of English; • Heterogeneous; • Membership defined by limited proficiency in English language use, which directly affects learningand assessment; • Membership is expected to be temporary.

9. Spanish predominant Large population Largest growth • Largest and fastest growing ELL groups: • Students who immigrated before Kindergarten • U.S. born children of immigrants More than 400 different home languages; Spanish predominant (70%) One of the fastest-growing groups among school-aged children in U.S. More than 9 million students, roughly 5.5 million of whom are classified as Limited English Proficient (LEP) Demographics By 2015, second-generation children of immigrants are expected to be 30% of the school-aged population

10. Learning challenges ELLs face unique learning challenges: • to develop the content-related knowledge andskills defined by state standards; • while simultaneouslyacquiringa second (or third)language ; • at a time when theirfirst language is not fully developed(e.g., young children); • to demonstrate their learning on assessments in English, their second language.

11. ELL Performance Outcomes • Some states have begun to look at the performance of ELLs on state tests after they have gained proficiency in English. • Although some reclassified ELLs do well, many students who have lost the formal LEP designation continue to struggle with: • academic text; • content-area knowledge; and • oral language skills.

12. Current Policy and Academic Achievement

13. English Language Learnersand the No Child Left Behind Act ELLs present unique challenges to: • Teachers, • Administrators, • Assessment systems, and • Accountability systems.

14. English Language Learnersand the No Child Left Behind Act NCLB: • High standards of learning and instruction for all students; • English Language Learners one of five areas of concentration to advance student achievement; • Increased awareness of the academic needs and achievement of ELLs; • Schools, districts, and states held accountable for teaching English and content knowledge to ELLs.

15. English Language Learnersand the No Child Left Behind Act Under NCLB, state education agencies are held accountable for the progress of ELLs in two ways: • Adequate Yearly Progress(AYP) expectations for reading and mathematics under Title I, and • Annual Measurable Achievement Objectives(AMAO) under Title III, demonstrating satisfactory progress in learning English and attaining English proficiency.

16. Academic Performance Indicators for ELLs On 4th grade National Assessment of Educational Progress (NAEP), ELLs were: • only1/4 as likely to score proficient or above in Reading as their native English speaking peers, and • only 1/3 as likely to score proficient or above in Math as their native English-speaking peers. Compared with native English-speaking peers, ELLs with a formal LEP designation are less likely to score “proficient” on state tests.

17. ELLs Taking Large-Scale Assessments:Participation and Valid Assessment

18. ELL Participation Rates • NCLB recognizes the importance of high participation to gain information about achievement and proficiency rates for this subgroup of learners. • ELLs are participating in large-scale assessments at much higher rates than in the past. • This raises awareness of the academic needs and academic achievement of ELLs as a distinct population. • Assessment results can be vital indicators of school performance and efficacy in serving specific subgroups.

19. ELLs & Large-Scale Assessments • It is not enough simply to have ELLs participate in large-scale assessments; participation must lead to valid inferences about their achievement and the effectiveness of educational programming. • Valid assessments are needed to inform instruction and program design: They provide detailed knowledge of ELL students’ weaknesses and strengths.

20. Large-Scale Assessments Used with ELLs • National Assessment of Educational Progress (NAEP); • State assessments of academic content; • State assessments of language proficiency.

21. NAEP Assessment • The only “nationally representative and continuing assessment” of subject area knowledge of U.S. students; • Does not score individual students or schools, but now allows some state-by-state comparison; • NCLB requires some schools to participate in NAEP grade 4 and 8 reading and math. (National Center for Educational Statistics, 2007)

22. State Assessments of Language Proficiency • NCLB requires annual assessment of ELLs’ English language proficiency with assessments aligned to K-12 ELP standards. These must include the 4 language domains: • Speaking and listening; • Reading and writing. • State ELP standards must align to the academic content standards and the challenging academic achievement standards set by the state for all students. (General Accounting Office, 2006)

23. State Assessments of Academic Content • Based on federally-mandated, state-defined standards; • Selected or developed by each state, with guidance from the U.S. Department of Education, Office of Elementary and Secondary Education. (August & Piche, 2002; General Accounting Office, 2006)

24. Content Knowledge & Language Proficiency • ELLs must direct more cognitive resources to processing the language of the test compared with fully English proficient students. • Assessments with the most linguistically challenging content show the largest performance gaps between ELLs and native English speakers. This is true for all domains, including math. • ELLs may also lack background knowledge that is central to understanding content, but may not be related to actual content knowledge. Effective accommodations allow students to demonstrate their knowledge of a concept by minimizing language obstacles without invalidating the assessment.

25. State Policies • Educational agencies across the nation provide accommodations to ELLs as needed; • The criteria for selection and strategies for implementation vary by state, according to many factors.

26. State Policies Rivera and colleagues (2006) A comprehensive taxonomy for thinking about accommodations, both direct and indirect linguistic support accommodations.

27. States Must Improve Accountability, Assessment, and Alignment for Education of ELLs Accountability • Schools are accountable for educational quality, both to those who fund them and to those whom they serve. Assessment • External assessment (state and district tests) and classroom assessment are intended to both reflect educational quality and enhance it. Alignment • There should be a match between external standards for quality and curricula and classroom practices (Menken, 2000).

28. Validity of Academic Knowledge Assessments for ELLs • ELLs must direct more cognitive resources to processing the language of the test compared with fully English proficient students; • ELLs may also lack relevant background knowledge that is central to understanding content, but may not be related to actual content knowledge.

29. Validity of Academic Knowledge Assessments for ELLs Using accommodations during testing may improve validity. • These would control for language and cultural factors. Valid assessments are also useful for instruction and program design. • They provide detailed knowledge of ELL students’ weaknesses and strengths. (Francis et al., 2006)

30. Consequences of Invalid Assessment • Does not inform the design of effective programs and curricula;. • May undermine quality and appropriateness of individual placement and instruction; • May limit a student’s academic achievement.

31. Possible Consequences of High Stakes Assessment • For schools performing poorly: • Federal funds withheld; • Schoolwide restructuring; • Student transfers allowed; • Teachers or administrators unrewarded. • For students performing poorly: • Additional resources, tutoring, supplemental services and school choice options. (August & Piche, 2002; Coltrane, 2002; Abedi et al., 2003; Francis, 2006)

32. Effective Accommodations for ELLs Taking Large-Scale Assessments

33. What are accommodations? Encompass changes to standard test administration procedures, including: • How the assessment is presented to the student; • How the student is allowed to respond; • Any equipment or materials to be used; • Extent of time allowed to complete the test; and • Changes to the environment in which the student takes the test.

34. Test Accommodations for ELLs • Use of accommodations during testing may improve validity of inferences based on test results. • For ELLs participating in large-scale tests, accommodations are modifications to the test or testing conditions designed to reduce the impact of limited English proficiency on the assessment of the target academic skill.

35. Test Accommodations for ELLs • Criteria for selection and strategies for implementation vary by state. • Appropriate accommodations will address ELLs’ linguistic needs either: • directly (e.g., language in test) or • indirectly (e.g., increased time to process language).

36. Types of Accommodations for ELLs Modification of the testing conditions: • Special scheduling, setting, or extended time; • Use of tools such as bilingual or English dictionaries or glossaries.* Modification of the test: • Directions or items may be read aloud in English or in native language; may be bilingual, native language, or simplified English version of test;* • Response options: response in native language; dictated response.* *Targets language

37. Characteristics of Appropriate Accommodations Take the student’s background into account: • Native language literacy, if test is to be written in native language; • Familiarity with the accommodation from classroom use, as with prior use of dictionaries; • Language of assessment and language of instruction need to match. Their use in combination must be guided by a specific rationale: • For example, use of dictionaries and extended time are a reasonable combination.

38. Accommodations of Testing Conditions Extended time* Breaks offered between sessions Bilingual glossaries* Bilingual dictionaries* English glossaries* English dictionaries* Accommodations as Test Modifications Directions read in English Directions read in native language Directions translated into native language Simplified English* Side-by-side bilingual version of the test* Native language test* Dictation of answers or use of a scribe Test taker responds in native language Partial List of Accommodations Responsive to Needs of ELLs * Denotes the accommodations examined in the meta-analysis

39. Effective Accommodations for ELLs: Results of a Meta-Analysis Research Findings

40. What is a Meta-Analysis? A meta-analytic review is a specific approach to research synthesis that attempts to quantify the effect of an intervention from a set of comparable studies.

41. Preliminary Research We examined the effect of 8 types of accommodations which, alone or combined, affected the performance of ELLs in large-scale assessments of achievement.

42. Most Common Accommodations for ELLs (GAO, 2006)

43. Effective Accommodations for ELLs: Results of a Meta-Analysis 11 studies in total: • Each study randomly assigned ELLs and non-ELLs to testing conditions with and without accommodations; • Involved 37 different samples of students; • Reported 37 different tests of the effectiveness of accommodations for ELLs.

44. Study Descriptions Grades included: • 11 studies with 4th graders; • 22 studies with 8th graders; • 2 studies with 5th graders; • 2 studies with 6th graders. Subject Areas: • 17 studies tested math skills; • 19 studies tested science skills; • 1 study tested reading skills.

45. Study Descriptions (cont’d) Type of test • 22 studies examined items from the National Assessment of Educational Performance (NAEP); • 6 studies examined items from the NAEP and the Trends in International Mathematics and Science Study (TIMSS); • 9 studies examined items from State Accountability Assessments (two different states).

46. Seven Accommodations Studied • Simplified English (15 studies) • English dictionary/glossary (11 studies) • Bilingual dictionary/glossary (5 studies) • Extra time (2 studies) • Spanish language test (2 studies) • Dual language questions (1 study) • Dual language booklet (1 study)

47. Accommodations Simplified English Involves linguistic changes in the vocabulary and grammar of test items to eliminate irrelevant complexity while keeping the content the same. • Some changes may be effected by eliminating non-content related vocabulary, shortening sentences, simplifying sentence structures where possible, using familiar or frequently used words, active instead of passive voice, and using present verb tense where possible.

48. Accommodations English dictionary/glossary Involves adding definitions or simple paraphrases for potentially unfamiliar or difficult words in test booklets (usually on the margins). • A variation on this accommodation is to provide computerized tests with built-in English glossaries. Typically, this latter variation involves a computer program that provides a simple and item-appropriate synonym for each difficult non-content word in a test.

49. Accommodations Bilingual dictionary/glossary ELLs are given access to dictionaries, glossaries, and marginal glossaries with words written in English and the student’s native language. • Another version of this accommodation uses computerized tests with built-in bilingual glossaries.

50. Accommodations Extra time • Providing more time than usual to complete test sections is among the most frequently used accommodations. • This accommodation does not change the test itself, but the testing conditions. • Extended time is usually provided in combination with other accommodations. • The rationale is to allow ELLs extra time to process the language of the test, or when bundling extra time with another accommodation, such as an English language dictionary, to allow time to use the other accommodation.