1 / 20

Designing More Accessible Achievement Tests for All Students

Designing More Accessible Achievement Tests for All Students. Stephen N. Elliott Learning Sciences Institute and Department of Special Education Vanderbilt University CCSSO 2009 National Conference on Student Assessment. Projects & Partners. CAAVES: Consortium for Alternate Assessment

aricin
Download Presentation

Designing More Accessible Achievement Tests for All Students

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Designing More Accessible Achievement Tests for All Students Stephen N. Elliott Learning Sciences Institute and Department of Special Education Vanderbilt University CCSSO 2009 National Conference on Student Assessment

  2. Projects & Partners • CAAVES: Consortium for Alternate Assessment Validity and Experimental Studies • USDE funded; 2006-2009 • Partners: AZ, HI, ID, & IN + Vanderbilt Measurement Group + Discovery Education Assessment • CMAADI: Consortium for Modified Alternate Assessment Development and Implementation • USDE funded; 2007-2010 • Arizona Dept. of Education • Indiana Dept. of Education • Visit Websites for Resources Discussed Today • http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml • http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml • http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home/TAMI_Project.xml CCSSO NSA Conference / Elliott 2009

  3. NCLB Act 2007 Amendments on AA-MAS Students with disabilities who exhibit persistent academic difficulties. Inattention Organizational difficulties Poor reading fluency History of below proficient test performances Low self-efficacy with testing Inclusive Testing & Better Results CCSSO NSA Conference / Elliott 2009

  4. Key Terms • Access: the opportunity for test-takers to demonstrate proficiency on the target construct of a test or a test item. In essence, complete access is manifest when a test-taker is able show the degree to which he/she knows the tested content. Access, therefore, must be understood as an interaction between individual test-taker characteristics and features of the test itself. • Accommodation: widely recognized in state testing guidelines as individualized changes to the setting, scheduling, presentation format, or response format of an assessment. • Modification: alterations or adjustments of test items to facilitate access for virtually all test takers. Appropriate modifications …. • Remove extraneous material, • Maintain the same depth of knowledge (DOK), • Do NOT change the grade-level construct being measured, and • Increase the validity of the inference from the test score. CCSSO NSA Conference / Elliott 2009

  5. Anatomy of an Item CCSSO NSA Conference / Elliott 2009

  6. CAAVES Procedures We completed the following….. Modified a common set of existing reading and math items to create items designed to be more accessible and still measure the same grade-level content as the original items. Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties. Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability. Conducted post-assessment survey of all students concerning their perceptions of item types and cognitive ease. CCSSO NSA Conference / Elliott 2009

  7. Guiding Theories & Research • Evidenced-based model of test score validity, • Universal design principles, • Cognitive Load Theory for designing instructional materials, and • Item writing research and practices. CCSSO NSA Conference / Elliott 2009

  8. Examples of Theory-Guided & Data-Based Item Modifications CCSSO NSA Conference / Elliott 2009

  9. Example: Original to Modified Item CCSSO NSA Conference / Elliott 2009

  10. Overview of Results Elliott, et al. (in press), Exceptional Children CCSSO NSA Conference / Elliott 2009

  11. Modifications Benefited all Groups Elliott, et al. (in press), Exceptional Children CCSSO NSA Conference / Elliott 2009

  12. Item Summary Reports: An Example CCSSO NSA Conference / Elliott 2009 12

  13. Estimating Impact Will AA-MASs result in more students with disabilities being considered “proficient” for AYP? We have explored the impact of some hypothetical cut scores for the CAAVES Reading and Math Scores. An actual Standard Setting is needed. Elliott, et al. (in press), Exceptional Children CCSSO NSA Conference / Elliott 2009

  14. Method for Documenting OTL AZ Cog Lab Study, 2008 CCSSO NSA Conference / Elliott 2009

  15. Cognitive Labs Excerpted from Kettler, Elliott, & Beddow, in press Peabody Journal of Education CCSSO NSA Conference / Elliott 2009

  16. Post-Assessment Focus Groups AZ CMAADI Pilot Study, 2009 CCSSO NSA Conference / Elliott 2009

  17. Evolving Modification Paradigm Step 1. Evaluate original item accessibility. Step 2. Reduce sources of construct-irrelevant variance in items. Step 3. Document changes to items. Step 4. Pilot test with student cognitive labs & post-assessment focus groups. Step 5. Field test with large sample of students. Step 6. Conduct psychometric & related analyses. CCSSO NSA Conference / Elliott 2009

  18. Characteristics of Appropriate Modifications Design Elements • Simply words and text structure • Delete extraneous words • Improve visuals and locate within item • Use bold text for important words • Eliminate least plausible distractor so there are 3 answer choices Desired Outcomes • Increase accessibility • Decrease item difficulty • Increase item discrimination • Increase reliability estimates • Reduce readability level w/i grade range • Maintain alignment w/ content stds. • Maintain DOK for all items • Increase validity of test scores • Reduce need for accommodations • Increase reading fluency • Improve students’ perceptions of tests & motivation to engage in testing CCSSO NSA Conference / Elliott 2009

  19. Colleagues’ Presentations • Quantifying and Improving Item & Test Accessibility – Peter Beddow, Vanderbilt • Using Students’ Insights to Influence Item & Test Design – Andrew Roach, Georgia State • Plausible Attractors & Item Psychometrics- Michael Rodriguez, University of Minnesota CCSSO NSA Conference / Elliott 2009

  20. Thanks! Thank you very much for your time and joining us for this session. Please provide follow-up questions and suggestions in writing to: Steve.elliott@vanderbilt.edu http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml CCSSO NSA Conference / Elliott 2009

More Related