Selecting Accommodations Wisely: Facilitating Test Access & Enhancing Implementation Integrity

1. Selecting Accommodations Wisely:Facilitating Test Access & Enhancing Implementation Integrity Stephen N. Elliott, PhD Dunn Family Professor of Assessment Department of Special Education Peabody College Vanderbilt University

2. Introduction & Background Information • Served on NRC Committee that authored Educating One & All, • Senior author of the Assessment Accommodation Guide, • Developed (w/ Braden) Assessing One & All online course & book, • Published 7 experimental studies (w/ a number of colleagues) on the effects of testing accommodations on test scores, and • Currently serve on Technical Work Groups for NAEP Accommodations & the IES National Alternate Assessment Study.

3. Presentation Outline • Words and Concepts Matter – let’s communicate! • Key Factors that Influence Accommodation Decision Making • 5-Step Decision Making Process & Documentation • Questions & Some Answers

4. Decisions about accommodations requires IEP teams to know... • The student’s abilities, • The instructional accommodations the student presently receives, • The test (content, item type, & testing conditions) and the skills measured by the test, • The state’s Testing Guidelines, • What it means to invalidate a test score, and • Previously used testing accommodations.

5. Michele - Case #1 Michele is an 8th grader who has had some difficulty learning and frequently exhibits poor work habits. She is functioning below grade level expectations in almost all subjects, but does not qualify for special services.

6. Tia - Case #2 Tia is a 4th grader who is classified as learning disabled. Her instructional reading level is 2nd grade, but she receive all her instruction in regular classes with some support from a consulting special education teacher. She has good listening and memory skills, and is a highly motivated student.

7. Ben - Case #3 Ben is chronologically a 10th grader who was diagnosed as autistic at age 3. Due to his pervasive communication difficulties, he receives much of his education in a highly structured special education classroom with 6 other students with developmental disorders.

8. Individual Students Have Individualized Needs • Most states require a student to have an IEP to be eligible for accommodation considerations. • Look into the classroom for evidence about accommodations needed to support learning. Many instructional accommodations generalize to the testing event. IEP Team Members Must Understand an Individual Student’s Instructional Support Needs before Making a Decision about Needed Testing Accommodations!

9. Definition & Functionof Testing Accommodations Definition - changes to the way a test is administered or responded to by a student. Such changes are often categorized as changes to the setting, timing, scheduling, presentation, and method of responding. Purposes • To facilitate participate from individuals who have not taken tests in the past. • To offset distortions in test scores caused by a disability without invalidating the test results (i.e., to increase the validity of the inference made from a test score). • To comply with IDEA and state regulations.

10. Testing Alterations:Further Defining an Accommodation Accommodation +------------------+ Modification • Unchanged Constructs • Individual Need • Differential Effects • Sameness of Inference Four attributes that differentiate an accommodation from a Modification. As one moves closer to a modification, the validity of the inferences from a test score becomes more suspect. Hollenbeck (2002)

11. Access Skills & Targeted Skills Testing accommodations facilitate access to a test and should reduce the error in test scores due to relatively poor access skills. Appropriate testing accommodations should not change or replace the skills that the test targets or is designed to measure. 2 metaphors: Eye glasses & Access Ramp

12. Target Skills Test items are designed to measure specific skills or abilities. For example, many mathematics items are intended to measure a student’s ability to reason, compute, and communication a solution or result. The skills or abilities that test developers intend the items to measure can be called target skills or abilities. The same mathematics items require a student to attend, read, remember some information, and ultimately respond by bubbling in an answer choice or writing an extended response.These latter skills are generally Not what the test developers designed the mathematics items to measure, but without these skills or abilities students can not access or interact with the test items to demonstrate whether or not they posses the target skills measured by the items. IEP Teams members must know what the test Measures!!!

13. Attending Listening Reading* Remembering Writing* Following directions Working alone Sitting quietly Turning pages 10. Locating test items 11. Locating answer spaces 12. Erasing completely 13. Seeing 14. Processing information in a timely manner 15. Working for a sustained period of time 16. Spelling* Key Access Skills*some access skills can also be target skills

14. Testing Accommodation Guidelines • Every state has some written guidance concerning allowable testing guidelines. • Most test publishers also have a generic set of testing guidelines that provide conservative guidance about the influence of testing accommodations on the validity of test score inferences (e.g.,http://www.ctb.com/media/articles/pdfs/general/guidelines_inclusive.pdf) IEP Team Members Must Know Their State’s Testing Accommodation Guidelines!

15. You Must Know the Test • It is critical that the IEP team be familiar with state standards and assessments before the test or assessments are administered. • The IEP team must have a working knowledge of the test format and what skills and knowledge are being measured by statewide assessments. • Knowledge of the test and student’s abilities drive decisions about participation and accommodations.

16. Technical Qualities of a Test Reliability: A reliable test provides you consistent results. That is, it is accurate over repeated uses. Validity: A valid test* measures what it says it measures. That is, it allows you to make an inference about a person’s knowledge or skill from his or her score on a given test. Reliability is a necessary, but not sufficient condition for validity!

17. Approaches to Interpreting Assessments • NRT (norm-referenced testing): Comparing people to people. • CRT (criterion-referenced testing): Comparing people to a standard of performance. Virtually all tests for NCLB use a CRT approach to interpretation. A CRT approach to test interpretation “tolerates” diversity in testing conditions as long as the construct being measured is unchanged!

18. Invalidation of a test score means that changes in the test presentation, administration, content, or response format have resulted in changes in what the test measures. Therefore, the inference one makes from a given test score is not justified or comparable to other scores.

19. The Interaction Effect: Validity Evidence • A common framework for interpreting the validity of accommodations has been to look for an interaction between educational status (students w/disabilities & students without disabilities) and accommodation conditions (accommodated & not accommodated). • The expectation has been that accommodations would improve the average score for a group of students with disabilities, but would have little or no affect on the average score of a group of non-disabled students.

20. H SWOD Test Scores SWD L No Acc Acc Illustration of the Interaction Paradigm

21. IEP Teams Must Understand Research Findings on Testing Accommodations • Most frequently used testing accommodation packages include extra time, shorten sessions, & reading support, • Typical effects of “appropriate” accommodation packages on test scores (ES = .40 for SWDs), • Questionable accommodations: Reading a reading test & use of calculators when mental computation is the target skill. • Testing accommodations have some unintended consequences too! • Implementation integrity is a growing concern.

22. 5-Step Systematic Decision-Making and Documentation Process The Assessment Accommodations Checklist provides a framework for IEP team’s accommodation selection, implementation planning, and documentation. • Step 1 – Complete Student Information • Step 2 – Meet with IEP Team to Select Accommodations • Step 3 – Communicate the Accommodation Plan with Parent(s) • Step 4 – Implement the Testing Accommodation Plan • Step 5 – Report and Evaluate the Use of the Testing Accommodations (Elliott, Kratochwill, & Schulte, 1999)

23. Dos & Don’ts for IEP Teams • Do systematically use accommodations during instruction and carry these into the assessment process. • Do base the decision about accommodations, both for instruction and for assessment, on the needs of the student. • Do consult the district or state list of approved accommodations after determining what accommodations the student needs. Then, reevaluate the importance of the accommodations that are not allowed. If they are important for the student, request their approval from the district or state. • Don’t introduce a new accommodation for the first time for an assessment. • Don’t base the decision about what accommodations a student will use on the student’s disability category. • Don’t start from the district or state list of approved accommodations when considering what accommodations a student will use in an upcoming test. Source: Thurlow, J. Elliott, and Ysseldyke (1998, pp. 61-62)

24. Making Assessment Decisions:A Summary • The IEP team makes decisions • Decisions are made on an individual basis • Decisions are documented in the IEP • Decisions are based on access needs; that is, accommodations are designed to affect deficits in access skills, not target skills. • Decisions about testing accommodations must not invalidate test results; the research literature provides some guidance, but a solid understanding of the concept of validity is needed • Decisions are made prior to the testing date • The testing event should be reviewed to determine success of accommodation effort and a record to guide future decisions documented.

25. Questions & Comments • I welcome your questions and reactions to any of the points I have made today. • I encourage resource sharing and will gladly provide you additional resources, if I have them.

26. Thank you! • I appreciate the opportunity to share my alternate assessment ideas and experiences with you. • Contact Information: Steve.elliott@vanderbilt.edu 615-322-2538