Accommodations Research & Universally Designed Assessments: Where We’ve Been & Where We’re Going

1. Accommodations Research & Universally Designed Assessments: Where We’ve Been & Where We’re Going Sandy Thompson Christopher Johnstone Amanda Blount Morse National Center on Educational Outcomes

2. Goals • Define and describe the purposes of test accommodations • Update and summarize what we know from research on accommodations use • Provide direction for future research on the effects of accommodations

3. Goals • Define “universally designed” assessments within the current context of large-scale assessments and accountability • Identify elements of universally designed assessments • Identify relevant research

4. Educational Accountability • Driven by federal law states have implemented statewide accountability systems. • One of the most challenging tasks has been determining appropriate ways to include students with disabilities in statewide assessments. • One of the most feasible ways to increase the participation of students with disabilities in assessments is through the use of testing accommodations.

5. Definition – changes in assessment materials or procedures Purposes To provide students with disabilities access to assessments, so that more can participate. Allow student’s to demonstrate their knowledge and skills to be assessed rather than the student’s disability. To comply with federal and state regulations. What are Assessment Accommodations?

6. Types of Accommodations

7. Accommodations:The National Picture • All states have accommodation policies. • States vary in terms of which accommodations they consider “standard” and “non-standard.” • Accommodation use is on the rise. • About 50% of students with learning disabilities received an accommodation during testing.

8. Summary of Research on the Effects of Test Accommodations • The first effort to summarize all currently available research on the topic of assessment accommodations was written by Tindal & Fuchs and published in 1999. • The current analysis was conducted to summarize research on the effects of test accommodations published from 1999 through 2001.

9. Summary of Research on the Effects of Test Accommodations • Forty-six research studies were reviewed. • All of the studies are empirical – include an analysis of data. • The studies have the following characteristics: • Purpose • Types of assessment, content areas, and accommodations • Participants • Research Design • Findings • Limitations

10. Purpose of the Research Most commonly cited purposes of research • Determine the effect of the use of accommodations on test scores of students with disabilities • Investigate the effects of accommodations on test score validity • Study institutional factors, teacher judgment, or student desirability of accommodation use • Examine patterns of errors across items or tests

11. Types of Assessment, Content Areas, and Accommodations • The majority of the studies used criterion referenced tests. • Mathematics was assessed in half of the studies, and reading/language arts was assessed in about one third of the studies. • Presentation accommodations were investigated most frequently. • “Oral presentation" was examined in almost half of the studies.

12. Participants in Reviewed Research • The number of participants ranged from 3 to nearly 21,000. • The largest number of studies included elementary school students. • Learning and cognitive disabilities were most frequently investigated among studies that documented participants types of disabilities.

13. Research Design • The studies were identified as representing one of four group research designs, a single subject research design, or a non-experimental or other design. • Over one third of the studies applied non-experimental or other designs to the study of accommodations effects.

14. Findings from Reviewed Research • Three accommodations showed a positive effect on student test scores across at least four studies: • Computer administration • Oral presentation • Extended time • However, additional studies on each of these accommodations also found no significant effect on scores or alterations in item comparability.

15. Findings from Reviewed Research • All of the meta analyses of accommodated conditions found a positive effect on scores. • All of the studies examining differential item functioning (DIF) found some items that exhibited DIF under accommodated conditions.

16. Limitations Cited in Reviewed Research Frequently cited limitations were: • Unknown variations among students included in the study • Sample sizes too small to provide adequate statistical support • Nonstandard administration of the accommodations across proctors and schools

17. Recommendations for Future Research Recommendations cited by researchers: • Replicating the research for validation and generalization • Investigating associations to specific disabilities • More detailed non-experimental studies be conducted to provide richer data • Increase researcher control of the testing process • Study larger groups of students

18. Conclusions Observations and recommendations: • Clear definition of the constructs tested • Greater clarity in accommodations needed by individual students • Desirability and perceived usefulness of accommodations by students themselves- the “End Users” • Consider Universal Design

19. Universal Design Research? • Pieced together from: • Specific accommodations research • Policy statements • Other fields (vision, ergonomics, graphic design, Universal Design of architecture research)

20. Universally designed assessments: • are designed from the beginning to be accessible and valid for the widest range of students • provide optimal standard assessment conditions

21. Who Benefits? • Universal design does not apply exclusively to people with disabilities or limited English proficiency • It applies to all individuals, with wide ranging characteristics

22. Think About Universal Design in Architecture and Tool Design • Curb cuts and ramps • Elevators that talk to you • Door handles rather than knobs • Special pen shapes that are easier to hold

23. Remember This? Official Ballot, Palm Beach County, Florida

24. Elements of UD Assessments • Inclusive assessment population • Precisely defined constructs • Accessible, non-biased items • Amenable to accommodations

25. Elements of UD Assessments(continued) • Simple, clear, and intuitive instructions and procedures • Maximum readability and comprehensibility • Maximum legibility

26. Preliminary Research in Universal Design • Sample of 230 students taken from four schools in US Southwest. • Two schools were “town” schools (pop. 20,000) and two were “rural” schools. • Students chosen from sixth grade teams that had populations of students with disabilities.

27. Research Design • Two tests were created, one from sample statewide test items, the other re-designed using UD principles. • Each student took both tests. • Students randomly assigned to take a particular test first to prevent practice effect. • Constructs held constant for each item.

28. Community Involvement • Advisory Board trained in principles of Universal Design and asked to comment / suggest improvements based on their perspectives. • Team consisted of three parents of children in special education program (one Navajo, one Latina, one Anglo) and one community member with dyslexia.

29. Sample Item Ramón is building a doghouse. He wants the roof of the doghouse to be at an angle that is more than 90° but less than 110°. Which angle below could he use for the roof? A. B. C. D.

30. Revised Item Which angle is more than 90° and less than 110°? A. B. C. D.

31. What Changed?? • Design element #2: construct more precisely defined. • Design element #3: bias eliminated (dog house, Ramón) • Design element #4: “built in accommodations” – un-timed, students circled answer on paper, did not bubble • Design element #5: simple instructions and procedures • Design element #6: more comprehensible language, underlined key words • Design element #7: larger font

32. Results • Means of two tests were compared and t-tests performed. • A difference of 8.16 (1.67 sig.) was found between means, a statistically significant finding. • Effect size calculated using Cohen’s d. Effect of design = .061 (or 6/10 Standard Deviation difference) – a “moderate effect”

33. Why? • Students with largest difference between two tests were interviewed to determine difference for them. • Students noted that: more direct language made it easier for them to “understand” items and unlimited time helped them to “think better” about items. Students also said they “remembered” content better on UD test.

34. “Simplex Sigillum Veri”“The simple is the seal of the true” Tests that remain true to constructs, are easy to understand, and contain language that is accessible to all will give the truest readings of what students do and do not know. Universal Design does not mean “dumbing down” a test.

35. What Have We Learned? • Design matters!! How a test is designed may effect how a student scores on that test. • Items that are better designed appear to aid students that are English Language Learners and with disabilities “show what they know” better. • This leads to more valid assessment of traditionally “under-performing” students.

36. Development Techniques for Universally Designed Assessments

37. “Think aloud” • Recently interviewed 90 students using think aloud protocol • 4th and 8th grade • Used multiple choice and constructed response items from state math test

38. Logistics • 10 researchers • 5 days • Worked in pairs • All sessions videotaped • Primary accommodations included oral administration and sign language interpretation

39. Student Characteristics

40. Overall Observations • Students who were confident of content did not have problems with design • Students who had no idea how to solve the problem did not have problems with design • Students “in the middle” – not sure of content, some reading difficulty, design made a difference

41. Examples of Student Perceptions • Many students didn’t see one of the cities on a map • The name of one of the cities was “Independence” - uncommon meaning • Box between top and bottom of item – some students did not read entire item • Sign for parallel gave away the answer • Some students read fraction 3 5/8 as “35 divided by 8” • Students unfamiliar with settings – “Glee club does number,” “fitness club”

42. Other Observations • Some students got a lot of “help” from sign language interpreters and teachers who wanted to make sure they understood the problem – what happens on test day? • Some student forms reported need for oral administration when they could clearly read the items independently – are some accommodations inappropriate? • For English language learners - is oral administration in English an appropriate accommodation on a Math test? • Should sign language interpreters have a script to follow so they don’t “give away” some answers?

43. Future Plans • Interview students with vision impairments • Partner with American Printing House for the Blind • Explore differential item analyses • Partner with researchers from CTB/McGraw-Hill • Produce short awareness video • Partner with Institute on Community Integration at University of Minnesota • Design and pilot training for item reviewers and item developers • Partner with Missouri Department of Education

44. More Information… NCEO Resources • Visit: http://education.umn.edu/nceo or Search for NCEO • Web site includes: • Topic introduction • Frequently Asked Questions • Online and Other Resources