Acknowledgements

1. Accessibility Issues of Scenario-Based Computer-Delivered Mathematics Items for Students with Disabilities and English Language Learners Teresa KingEducational Testing ServiceNational Conference on Student Assessment June 20, 2011

2. Acknowledgements • Mike Wagner • Karen Harris • Peggy Redman • Malcolm Bauer • Joni Lakin • Maria Martiniello • Cara Laitusis • Lucy Amato • Mike Eckert

3. Overview • What is CBAL? • Purpose of current research study • Interview findings by student group and assessment features • Validity implications • Introduction of 2011 accessibility research study

4. CBAL • Cognitively Based Assessment of, for, and as Learning • Program of research intended to create a model for a balanced K-12 assessment system that: • Documents what students have achieved (“of learning”) • Helps identify how to plan instruction (“for learning”) • Is considered by students and teachers to be a worthwhile educational experience in and of itself (“as learning”) • Has conceptual, summative, formative, and professional support components • Prototype development has focused primarily on middle-school reading, writing, and mathematics

5. Purpose • Examine the accessibility of the linguistic complexity, the test layout, and the technology features of Grade 8 Cognitively Based Assessment of, for, and as Learning (CBAL) summative mathematics items

6. An item appeared here that assessed a student’s understanding of the linear relationship between time and distance traveled, as represented in a table. The item featured an online tool that the student could access to illustrate the time/distance relationship, as well as an online calculator, and a numeric-entry response format.

7. An item appeared here that assessed a student’s understanding of similarity of geometric shapes or figures. The item featured an online protractor and online calculator that could be accessed by the student, and a mixed response format that included both text entry and numeric entry.

8. Materials – CBAL test • Sections of three CBAL (PAA) summative Grade 8 Mathematics item sets were selected • Items were selected to have various linguistic features, simulations and tools • Linguistic features • E.g., Directions embedded in 1st slide or items with separate directions screen, lengthy descriptions or shorter sentences

9. Tools and Simulations • Tools - Calculator, protractor, slider • Simulation - Moving sidewalk • Precise mouse manipulations – clicking on a graph to draw a line, rotating tiles, dragging tiles, using a protractor • Response format - enter a # or text into a text box, selecting radio button, select response in a drop down box

10. Mathematics Knowledge • Lines of symmetry • Rotation • Reflection • Concept of similar figures when one figure is an enlargement of the other or when all the corresponding angles are the same • Determining speed when the formula is given • Slope • Find slope from graph • Interpretation of slope • Y-intercept • Rate of change

11. Students • Native Spanish-speaking ELL students • Students with a disability that affects their motor skills • Comparison group of students • Native English speaking students • No learning or physical disabilities

12. Research Questions • Are CBAL math assessments fair for students who are ELLs and for students who have fine-motor disabilities? Specifically, which aspects of the test instructions and tasks may include construct-irrelevant factors affecting the performance of these groups. • What challenges do ELLs find in comprehending and communicating their knowledge in CBAL summative tasks? How are these challenges different for ELLs than for non-ELLs? • How can we increase the accessibility and validity of assessment for ELLs and for students who have fine-motor disabilities regarding test design and administration with accommodations?

13. Procedure • Introduction/training of the CBAL layout, tools, and demonstration of thinking aloud • Cognitive Interviews • One-on-one administration • ELL interviews conduced by a trained bilingual Spanish/English interviewer

14. Cognitive Interview Details • Students think aloud while reading instructions and questions • Retrospective questions followed each screen of questions • ELL and SWD specific questions • Generic questions

15. Spanish Interviews • Directions and follow up questions could be given in English or Spanish • The text of the test items, answer options, and directly related test questions were only permitted in English • Thoughts spoken aloud could be in English or Spanish • Test responses = English only

16. ELL Findings • All students were verbal, all but one student preferred to speak in Spanish and one student said it was difficult to think aloud and answer

17. ELL Findings • No technology issues observed • No students tried to use keyboard shortcuts • No students had issues with making precise manipulations with the mouse • Mistakes or misconceptions may be eliminated with more directions or by providing Spanish directions

18. Linguistic Findings • One ELL did not know the word “tiles” • Many ELLs did not know the concept of slope or what the word assumption meant

19. SWD Findings • Extra time was imperative • Students demonstrated perseverance • Some observed difficulty interacting with a graph with small lines • Students were able to click where they intended but it required more effort

20. An item appeared here that assessed a student’s understanding of similarity of geometric shapes or figures. The item featured an online protractor and online calculator that could be accessed by the student, and a mixed response format that included both text entry and numeric entry.

21. Accessibility Findings • No students tried to use keyboard shortcuts • No major issues with any of the tools or simulations • Suggestion by some students to modify the slider tool • Most response formats were not found to be problematic; students knew how to interact and respond • Enlarging the text may be helpful

22. Validity Implications • Extent of the linguistic challenges posed by content assessment items impact ELLs’ interactions with and performance on them • Specific wording issues • ELP correlated with ability to interact appropriately with the test • Further examine language support options, simplified text, and changes to the test directions

23. Validity Implications • Extent of the technology features impact students with motor disabilities impact their interactions and item performance • Students were able to demonstrate what they know; however they need more time to take the test • Option to enlarge the test material would be helpful • Additional research is underway

24. 2011 Current Research • Use the principles of universal design to make existing items more accessible for students with disabilities and English language learners • Selected revisions examined for 3 similar student groups • Native Spanish speaking ELLs • Students with learning disabilities • Comparison students

25. Language Modifications User-InterfaceModifications • Read aloud of all items and directions • Spanish translations • Language simplification • Scribe for extended responses • Warnings for incomplete items • Increased white space • Add/remove graphics • Scratchpad for note-taking • Access to tool instructions throughout item

26. Preliminary Accessibility Thoughts • Consult with item developers to ensure the additions or changes do not affect the construct being measured • Work early and closely with bilingual test developers to allow proper consideration of linguistic features of test items • Avoid terms that are lengthy to translate or cannot be translated • Awareness of cognates that may benefit or confuse ELLs when translated

27. Thank you! Email: TKing@ets.org CBAL information: http://www.ets.org/research/topics/cbal/initiative