Applying Principles of Universal Design to Assessment Item Modification

1. Applying Principles of Universal Design to Assessment Item Modification Peter A. Beddow III Vanderbilt University Nashville, TN June 2008

2. Background • Definition of Universal Design: • The design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design. • (Center for Universal Design, NC State University, 2008; http://www.design.ncsu.edu/cud/about_ud/udprincipleshtmlformat.html#top) CAAVES Item Modification Study 2008

3. Universal Design (cont.) Equitable use Flexibility in use Simple and intuitive Perceptible information Tolerance for error Low physical effort Size and space for approach and use (Center for Universal Design, 2008) CAAVES Item Modification Study 2008

4. Principles of Universal Design are intended to maintain equity among user groups while providing the same or equivalent means of use for all users; Ideally, services, products, or systems employing universal design principles: Eliminate unecessary complexity; Arrange information consistent with its importance; Use prompting and feedback during and after task completion; Present information redundantly (visual, verbal, tactile); Maximize legibility and construct; Simplify directions and instructions; Reduce the potential for error. Universal Design (cont.) CAAVES Item Modification Study 2008

5. Item Accessibility and Modification Guide An Instrument for Designing and Modifying Assessment Items

6. Purpose of the Instrument • The Item Accessibility and Modification Guide (Beddow, Kettler, & Elliott, 2007) was designed to facilitate the analysis of test item features to enhance meaningful responses from, and provide equal access for, all students; • The instrument can be used: • To aid in examination and modification of existing items; • To influence the design of new items; or • As a training tool for item writers. CAAVES Item Modification Study 2008

7. Influence • The Item Accessibility and Modification Guide was directly influenced by National Center for Educational Outcomes publications, specifically: • NCEO’s Policy Directions 18 (Johnstone, Thurlow, Moore, & Altman, 2006) which provides expert review recommendations for test item. • Section 508 guidelines. • To date there has been no effort to provide a rating instrument to assist in the consideration of universally designed assessment items. CAAVES Item Modification Study 2008

8. Organization • The Item Accessibility and Modification Guide consists of two sections: • Considerations for Universally Designed Assessment Items (7 categories, 38 items); • Computer-based Tests: Considerations for Universally Designed Items (5 categories, 26 items). CAAVES Item Modification Study 2008

9. Procedure • The rater is asked to record the item being reviewed, his/her name, and a brief description of the construct the item is intended to measure. • For each accessibility item, the rater is asked to check Yes or No (or N/A if the question is not applicable to the target item). • For each category, the rater is asked to circle the number that best represents his/her summative evaluation of the item’s accessibility for that category based on checklist responses: • 0 = The item is Not Accessible • 1 = The item is Minimally Accessible • 2 = The item is Moderately Accessible • 3 = The item is Maximally Accessible • The Total Item Accessibility Rating is found by totalling overall evaluation ratings for the item across categories. • It should be noted that the overall rating is intended as a simple way to categorize items by high, medium, or low accessibility, as opposed to providing a definitive indicator of accessibility that is based on research data. CAAVES Item Modification Study 2008

10. Considerations for Universally Designed Assessment Items • 7 categories: • Content Relevance (2 items); • Text Content (10 items); • Text Economy/Conciseness (3 items); • Text Appearance (7 items); • Visual Content (if applicable; 7 items); • Fairness for Subgroups (8 items); and • Format Flexibility (5 items). CAAVES Item Modification Study 2008

11. Considerations for Universally Designed Assessment Items CAAVES Item Modification Study 2008

12. Considerations for Computer-based Test Items • 5 categories: • Layout and Design (6 items); • Navigation (5 items); • Screen Reader Considerations (5 items); • Test Specific Options (4 items); and • Computer Capabilities (6 items). CAAVES Item Modification Study 2008

13. Considerations for Computer-based Test Items CAAVES Item Modification Study 2008

14. The Item Accessibility and Modification Guide • The instrument is available electronically at: http://www.vanderbilt.edu/lsi/caavespage/Item_Accessibility_and_Modification_Guide.pdf • Manuscript: http://www.vanderbilt.edu/lsi/caavespage/Item_Accessibility_and_Modification_Guide_Manuscript.pdf CAAVES Item Modification Study 2008

15. Thanks for listening. Peter A. Beddow III peter.beddow@vanderbilt.edu CAAVES Item Modification Study 2008 CAAVES Item Mod Data 2008