Creating Innovative Tests: Applying Universal Design to Assessment Practices

1. Creating Innovative Tests: Applying Universal Design to Assessment Practices Assessment Colloquium November 30, 2007 Manju Banerjee, Ph.D. Assistant Professor in Residence Special Education

2. Just imagine --- If there were no tests, no assessment, no accountability as we know it? Student perspective: Teacher perspective: Policy maker perspective:

3. “Opportunities borne of new technologies, desires borne of new understandings of learning ---- a new generation of assessment beckons. To realize the vision, we must reconceive how we think about assessment, from purposes and designs to production and delivery.”(Mislevy, Steinberg, & Almond, 1999, p.6)

4. Computer-based tests (CBT) are the “next frontier” in high stakes assessment(Thompson, Johnstone, & Thurlow, 2002)

5. What is the appeal of computer-based tests? Opportunity to create tests that support accessibility needs of diverse test takers -- Universal Design (UD) UD is anchored in the belief that a design that works well for examinees with disabilities, improves usability for all individuals (Center for an Accessible Society, 2006) • What is universal design? (Center for Universal Design, 1997) • What makes a test universally designed? • * Seven Elements of a universally designed test (Thompson, Johnstone, & Thurlow, 2002)

6. Application of Universal Design to High Stakes Tests Maximum usability Widest range of consumers Without design adaptations Minimize construct irrelevant features Disabilities, ELL, Non-traditional age Built-in from the start Include test taking features EXAMINEE CHOICE • Examinee choice is “flexibility to access and express in • the mode or methods that best suit the individual”(Hall, 2005, p. 2) • (Russell, Goldberg, & O’Conner, 2003)

7. 1. Objective of Study • Inform product development of high stakes tests • Based on current research on features that support “examinee choice” in high stakes test design Test taking tools On-screen item display tools Access tools • Bridgeman, Lennon, & • Jackenthal, 2002 • Mazzeo & Harvey, 1988 • Pommerich, 2004 • Pommerich & Burden, 2004 • Goldberg & Pedula, 2002 • Peak, 2005 • Lunz & Bergstrom, 1994 • Vispoel et al., 2000 • Mandinach et al., 2005 • Sireci, Li, & Scarpati, 2003 • Tindal & Fuchs, 2000

8. II. Background Information Features of Examinee Choice • Tindal, 1998 • CTB/McGraw-Hill, 2004 Construct neutral Construct related Test taking tools Access tools On-screen item display

9. II. Background Information (Cont.) • U D increased accessibility for all examinees • Accessibility is maximized when examinees have choice over features of test design • Research on features of test design fall into three broad categories: • (1)Test taking tools (2)Item Display (3) Access tools • Some features are construct neutral/construct irrelevant; others are construct related (including test accommodations) • Allowing examinees to choose features of test design based on individual preferences needs to be explored for a wide range of features including features that affect test construct • U D suggest a framework but research is still emerging on the application of UD to high stakes CBTs.

10. III. Methodology and Procedures Research questions

11. III. Methodology and Procedures (cont.) Research Design, Instrumentation, Pilot Study

12. III. Methodology and Procedures (contd.) Instrumentation – Test Features

13. III. Methodology and Procedures (contd.) Instrumentation

14. III. Methodology and Procedures (cont.) Instrumentation

15. III. Methodology and Procedures (contd.) Instrumentation- Creating the 1st choice exercise • Given 4x3x7 (features) = 84 combinations • Select a unique group of 4 from 84 combinations

16. III. Methodology and Procedures (contd.) Data Analysis Research Question 1 Research Question 2 Rank-ordered choice exercise data Voluntary top feature choice exercise data Rank-ordered Logit Regression Multinomial Logit Regression

17. III. Methodology and Procedures (contd.) Data Analysis – Rank-ordered logit regression

18. III. Methodology and Procedures (contd.) Data Analysis – Rank-ordered logit regression

19. III. Methodology and Procedures (contd.) Data Analysis – Multinomial logit regression

20. IV. Results – Participant demographics

21. IV. Results - Model 1 *p<0.10, **p<0.05, ***p<0.01

22. IV. Results - Model 1 (contd.) *p<0.10, **p<0.05, ***p<0.01

23. IV. Results - Model 2 *p<0.10, **p<0.05, ***p<0.01

24. IV. Results - Model 2 (contd.) *p<0.10, **p<0.05, ***p<0.01

25. IV. Results - Model 3 (contd.)

26. IV. Results - Model 3 (contd.)

27. IV. Results (contd.) Test of equality of regression coefficients for LD/ADHD status

28. IV. Results -Voluntary Top Feature Choice Exercise

29. V. Summary of Results and Discussion

30. V. Summary of Results and Discussion (contd.)

31. V. Summary of Results and Discussion (contd.)

32. V. Implications for Future Research

33. V. Limitations of Study

34. ***** END OF PRESENTATION