Universal Design of Assessments

1. Universal Design of Assessments Christopher Johnstone, National Center on Educational Outcomes National Center on Educational Outcomes

2. Goals • Define “universally designed” assessments within the current context of large-scale assessments and accountability • Identify elements of universally designed assessments • Identify relevant research National Center on Educational Outcomes

3. Universal Design Research? • Pieced together from: • Specific accommodations research • Policy statements • Other fields (vision, ergonomics, graphic design, Universal Design of architecture research) National Center on Educational Outcomes

4. Universally designed assessments: • are designed from the beginning to be accessible and valid for the widest range of students • provide optimal standard assessment conditions National Center on Educational Outcomes

5. Who Benefits? • Universal design does not apply exclusively to people with disabilities or limited English proficiency • It applies to all individuals, with wide ranging characteristics National Center on Educational Outcomes

6. Think about universal design in architecture and tool design • Curb cuts and ramps • Elevators that talk to you • Door handles rather than knobs • Special pen shapes that are easier to hold National Center on Educational Outcomes

7. Remember this? OFFICIAL BALLOT, PALM BEACH COUNTY, FLORIDA National Center on Educational Outcomes

8. Elements of UD Assessments • Inclusive assessment population • Precisely defined constructs • Accessible, non-biased items • Amenable to accommodations National Center on Educational Outcomes

9. Elements of UD Assessments (continued) • Simple, clear, and intuitive instructions and procedures • Maximum readability and comprehensibility • Maximum legibility National Center on Educational Outcomes

10. Preliminary Research in Universal Design • Sample of 230 students taken from four schools in US Southwest. • Two schools were “town” schools (pop. 20,000) and two were “rural” schools. • Students chosen from sixth grade teams that had populations of students with disabilities. National Center on Educational Outcomes

11. Research Design • Two tests were created, one from sample statewide test items, the other re-designed using UD principles. • Each student took both tests. • Students randomly assigned to take a particular test first to prevent practice effect. • Constructs held constant for each item. National Center on Educational Outcomes

12. Community Involvement • Advisory Board trained in principles of Universal Design and asked to comment / suggest improvements based on their perspectives. • Team consisted of three parents of children in special education program (one Navajo, one Latina, one Anglo) and one community member with dyslexia. National Center on Educational Outcomes

13. Sample Item Ramón is building a doghouse. He wants the roof of the doghouse to be at an angle that is more than 90° but less than 110°. Which angle below could he use for the roof? A. B. C. D. National Center on Educational Outcomes

14. Revised Item Which angle is more than 90° and less than 110°? A. B. C. D. National Center on Educational Outcomes

15. What changed?? • Design element #2: construct more precisely defined. • Design element #3: bias eliminated (dog house, Ramón) • Design element #4: “built in accommodations” – un-timed, students circled answer on paper, did not bubble • Design element #5: simple instructions and procedures • Design element #6: more comprehensible language, underlined key words • Design element #7: larger font National Center on Educational Outcomes

16. Results • Means of two tests were compared and t-tests performed. • A difference of 8.16 (1.67 sig.) was found between means, a statistically significant finding. • Effect size calculated using Cohen’s d. Effect of design = .61 (or 6/10 Standard Deviation difference) – a “moderate effect” National Center on Educational Outcomes

17. Why?? • Students with largest difference between two tests were interviewed to determine difference for them. • Students noted that: more direct language made it easier for them to “understand” items and unlimited time helped them to “think better” about items. Students also said they “remembered” content better on UD test. National Center on Educational Outcomes

18. What have we learned? • Design matters!! How a test is designed may effect how a student scores on that test. • Items that are better designed appear to aid students that are English Language Learners and with disabilities “show what they know” better. • This leads to more valid assessment of traditionally “under-performing” students. National Center on Educational Outcomes

19. “Simplex Sigillum Veri”“The simple is the seal of the true” Tests that remain true to constructs, are easy to understand, and contain language that is accessible to all will give the truest readings of what students do and do not know. Universal Design does not mean “dumbing down” a test. National Center on Educational Outcomes

20. More information? NCEO Resources • Visit: http://education.umn.edu/nceo or Search for NCEO • Web site includes: • Topic introduction • Frequently Asked Questions • Online and Other Resources including synthesis report and policy directions for Universally Designed assessments. National Center on Educational Outcomes