Adding Value to Accommodations Decision-Making (AVAD)

1. Adding Value to Accommodations Decision-Making (AVAD) 2006 Enhanced Assessment Instruments Grant Project Elizabeth Jones and Christopher Webster, South Carolina Department of Education

2. 2006 Enhanced Assessment Instruments Grant Award The South Carolina Department of Education received a 2006 Enhanced Assessment Instruments Grant award to undertake the AVAD project.

3. Overall goal of AVAD: To ensure the valid, effective use of accommodations designed to increase access to large-scale assessments for English language learners and students with disabilities

4. AVAD Project Partners • South Carolina Department of Education, lead state and fiscal agent • LEP SCASS • ASES SCASS • Michigan State University: Rebecca Kopriva • University of Oregon: Gerald Tindal • University of Nebraska/Psychometric Inquiries: Barbara Plake • Phoebe Winter, project evaluator

5. AVAD Project Period • Current project period from October 1, 2006 to March 31, 2008 • One-year no-cost extension will be requested • SCASS grant activities will begin in spring/summer 2007 • SCASS grant activities will be scheduled during or immediately before/after regularly scheduled SCASS meetings

6. AVAD’s Objectives: 1) to validate and enhance the Selection Taxonomy for English Language Learner Accommodations (STELLA) and publish the program on the SDE’s Web site 2) to create and validate a decision-making taxonomy for the Accommodation Station (AS) based on the ASES Accommodations Manual and publish the program on the SDE’s Web site

7. AVAD’s Objectives, cont.: 3) to develop 20 prototypes of computer-based access enhanced science items for grade four, and 4) to disseminate results through reports, the SDE’s Web site, User and Technical Manuals for STELLA and AS, the ASES Professional Development Guide video, and Web-Ex teleconferences

8. What role does the LEP SCASS play? • LEP SCASS will serve as expert panel to provide input and feedback on STELLA and computer-based access-enhanced prototype items • Opportunity to participate in a subgroup of states within the SCASS focusing on accommodations issues • Accommodations subgroup will commit to participation in AVAD meetings/conference calls • Updates on AVAD will be provided to the larger SCASS group • (ASES SCASS will play a similar role in Accommodation Station design and development)

9. What is STELLA? STELLA=Selection Taxonomy for Testing English Language Learners • an online system to assist educators in assigning testing accommodations to English language learners (ELL) so that individual students receive appropriate accommodations • developed by a 2003 Enhanced Assessment Instruments grant

10. STELLA Overview Data Collection • Teacher Form • Parent Form • Records Form Preloaded in System • Student Information Variables • Test Accommodations • Conversion and Consolidation Rules • Decision-Making Rules Output • Student Profile • Accommodation Decisions for Each Student Associated Materials • Data Collection Manual • Output Interpretation Manual • Technical Manual

11. Current Accommodations loaded in STELLA system • Pre-Test Best Practice Accommodations • Family Assessment Night • Tailored Classroom Support • Forms • Standard or some Universal Design Forms • Access-based Form • L1 or Side-by-Side forms as available • Tools • Bilingual word list, general or test specific • Picture-word dictionary • Problem solving tools

12. Current Accommodations loaded in STELLA system • Administration • Small group • Individual Administration • Oral English • Oral L1 • Language Liaison • Extra time • More frequent breaks • Response • Written L1 or code switching • Oral English • Oral L1 or code switching • Demonstrated or modeled response

13. STELLA Screenshots What does STELLA look like?

14. Records Form Language of instruction English language proficiency test L1 proficiency test

15. Records Form Type of ELL program

16. Parent/Guardian Form L1 Proficiency in four domains

17. Attendance in U.S. schools Schooling in native country

18. Classroom equipment, books, and supplies in native country Types of assessments in native country

19. Grading and assignments Test scores and experiences with testing

20. Teacher Form • English and L1 proficiency • (L1 includes ‘don’t know’ option) • Standardized score accuracy and judgments about reasons for inaccuracy • Student’s experience with standard test formats • Student’s perceived purpose of standardized testing • Classroom test condition options • Condition options that help student on classroom tests, evaluations

21. Experience with standardized testing Testing conditions used in the classroom

22. Beneficial testing conditions in the classroom

23. System Output • Individual Student Profile • English language proficiency, 4 domains • L1 proficiency, 4 domains • Cultural proximity, 3 variables: • Time in U.S. • Experience with testing procedures • Schooling proximity, native country • to U.S. experience

24. System Output: STELLA, Teacher Recommended and SEA/LEA Allowed Since different states allow different accommodations, STELLA program output is set up to identify the subset of allowable accommodations for each student. Non allowable accommodations selected for each student based on research and other best practice literature are also displayed in gray. This provides guidance to teachers and others about additional accommodations that could be useful for ELL students with specified needs.

25. Test Conversion Rules Currently, output from 4 English language proficiency tests (including ELDA) is preloaded. There is also the opportunity for personnel to add results from another test. In all cases score conversion rules place students on a common scale with four levels: • Beginner ELL • Low intermediate ELL • High intermediate ELL • Grade-level-competitive ELL

26. Student Information ConsolidationRules In several cases, more than one piece of student information is used to make judgments about his or her level on relevant variables. Two types of consolidation rules are part of STELLA. These are: • Consolidation of data from related items • Consolidation of information from more than one source

27. Combining Data from More Than One Question: An Example Time in School = LOW If the student has been in the US less than 1 academic year. OR If the student has been in US between 1 and 2 years AND has missed more than two months of school per year for 1 or more years. OR If the student has been in US between 2 and 3 years AND has missed more than two months of school per year for more than one year.

28. Combining Data from More Than One Source: An Example Consolidation rules are also used to determine L1 proficiency and English language proficiency. For L1 proficiency, consolidation rules are applied to information from teacher, parent and records forms.

29. Parents describe student as performing above grade level in L1 Teacher describes student as performing below grade level in L1 Parents trump teacher’s knowledge of L1, but test scores of L1 proficiency trump parents. What Trumps What? Consolidation Rules Example

30. Test Accommodation Decision-Making Rules A beginning set of decision-making rules were developed and tested. These rules take relevant student information and pair it with relevant accommodation factors for individual students. The accommodation factors identify which student needs the specific accommodation was designed to remediate. In this way, individual students are matched with accommodations appropriate to their particular needs.

31. LEP SCASS Validation and Enhancement of STELLA Activities • The LEP SCASS accommodations sub-group will validate existing accommodations decision-making rules by participating in a two-meeting and online expert panel process led by Barbara Plake. • Profiles of ELL students will be matched with specific sets of accommodations; accommodations for individual student profiles will be rated for validity and effectiveness; results will be compared to existing STELLA accommodations decision-making rules. • STELLA decision-making structure will be tweaked based on results of expert panel process. • STELLA interface and forms will be reviewed and revised.

32. Computer-based access-enhanced (CAE) science items review • CAE items aim to reduce barriers to access that English language learners may encounter in typical large-scale assessment items without altering the construct being measured or the rigor of the items • South Carolina will develop 20 prototypes of computer-based access-enhanced science items for grade four and place them online • LEP SCASS will review the CAE items and provide feedback

33. Benefits of Participation • LEP SCASS member states will have input into the design and development of STELLA • LEP SCASS member states (and any other interested parties) will have free access to the STELLA once it is completed and placed on the SDE Web site • LEP SCASS member states will be able to adapt the STELLA to best fit the policy and needs of their states

34. ASES SCASS: Parallel Activities • The ASES SCASS will serve as an expert panel for the development and review of the Accommodation Station (AS), an online decision-making model designed to assist IEP teams in assigning valid and appropriate accommodations to individual students with disabilities. • Current AS collects and consolidates information but does not include a decision-making algorithm. • The ASES manual and materials will serve as the basis for an algorithm that will be built and incorporated into the current AS. • A professional development video component will be developed to accompany existing ASES materials. • Sharon Hall of ASES will share additional details.

35. Questions? Please contact Therese Gleason Carr at tcarr@ed.sc.gov or (803) 734-3747 Thank you very much for your collaboration!