The Use of Accommodations for a K-12 Standardized Assessment: Practical Considerations from a State Perspective 1999 thr

1. The Use of Accommodations for a K-12 Standardized Assessment:Practical Considerations from a State Perspective 1999 through 2004 Patricia Almond, University of Oregon     Meagan Karvonen, Western Carolina University March 21, 2006 Savannah, GA Almond, Karvonen Savannah, GA March 2006

2. Although SWD often take state tests with accommodations States often have a hard time determining which accommodations to allow . . .(Olson, 2004) Almond, Karvonen Savannah, GA March 2006

3. Accommodations from a State PerspectiveIn the Context of Requirements and Responsibilities IDEA/NCLB State Legislation State Policies (testing, promotion, graduation, etc.) Training and Guidelines Assessment Technical Adequacy Almond, Karvonen Savannah, GA March 2006

4. Multiple Purposes and Full Participation “To earn their keep, state assessment systems attempt to do it all, often by asking the same assessment to fulfill several purposes.” (Almond, Lehr, Thurlow, Quenemoen, 2002, p. 346) Almond, Karvonen Savannah, GA March 2006

5. Purposes that can be applied to large-scale assessment: • Aiding in instructional decisions about individual students • Providing information about the status of the education system • Motivating change • Evaluating program • Holding schools accountable for student performance • Encouraging change in classroom instruction • Certifying individual student achievement and mastery (Heubert and Hauser, 1999) Almond, Karvonen Savannah, GA March 2006

6. School and District Accountability Schools are effective or in need of improvement High stakes rewards and sanctions will cause improved achievement Individual Student Proficiency Promotion to next grade Graduation from high school Certification Types of decisions based on results obtained from assessments taken with accommodations: Almond, Karvonen Savannah, GA March 2006

7. Standard 13.2 the Standards In educational settings, when a test is designed or used to serve multiple purposes, evidence of the test’s technical quality should be provided for each purpose. Almond, Karvonen Savannah, GA March 2006

8. Aligning the Newly Reauthorized IDEA with Title 1 . . . States have an affirmative obligation to determine what types of accommodations can be made to assessments while maintaining their reliability and validity House Committee Report on IDEA reauthorization? (108-77, April 29, 2003) Almond, Karvonen Savannah, GA March 2006

9. State Responsibilities • Documentation for Peer Review • Written policies, lists of allowable accommodations, score reports, empirical studies • Training for educators on how to select appropriate accommodations that will not invalidate the scores - Peer Review Guidance (2004) Almond, Karvonen Savannah, GA March 2006

10. State Policies on Accommodations • Differentiating between accommodations (allowable, standard) and modifications (not allowable, nonstandard) • Variations in documentation policies, reporting practices, eligibility guidelines (IEP, 504, ELL, universal) • Some decision making criteria: • Use of instructional accommodations (sometimes used for period of time prior to assessment) • Accommodation does not provide unfair advantage/change validity • Prohibitions against decisions based on program setting, disability category • Thurlow, Lazarus, Thompson, & Morse, 2005 Almond, Karvonen Savannah, GA March 2006

11. Potential problems in decision-making(school level) • Offering more (or fewer) accommodations than might be needed • Mismatch between instructional and assessment accommodations • Lack of documentation about decisions made • Lack of knowledge about accommodation vs. modification Almond, Karvonen Savannah, GA March 2006

12. Guidance from the Standards • Those who make accommodation decisions should be aware of existing research on the effects of disabilities on performance (10.2). • Decision-makers need to have access to accurate and current information about potential accommodations and modifications (10.8). • Accommodations and their rationales need to be described in detail (in administration manual and technical documents), including possible limits on validity of inferences (10.4, 10.5) Almond, Karvonen Savannah, GA March 2006

13. Progress, and room for growth… • Increased use of accommodations in large-scale assessments (Ysseldyke et al., 2004) • States recently identified as current and emerging issues related to accommodations: • Determination of appropriate and allowable accommodations • Technical assistance on appropriateness of specific accommodations for both instruction and assessment • Professional development, especially for general education teachers • Degree of specificity required in state guidance -NCEO 2005 State Report Almond, Karvonen Savannah, GA March 2006

14. In the Opening Session we received a report on the status of accommodations: Accommodations in State Policies-What a Wonderful World of Diversity: Issues and Implications     Martha Thurlow, NCEO Almond, Karvonen Savannah, GA March 2006

15. In this Session We ProvideA Case Study: One State’s Approach to Determining “Allowable” Testing Accommodations From 1999 through 2004 Almond, Karvonen Savannah, GA March 2006

16. Chronology Almond, Karvonen Savannah, GA March 2006

17. Comprehensive notAlternate • Integral part of the whole assessment system • Inclusive of all students Almond, Karvonen Savannah, GA March 2006

18. Knowledge & Skills Reading/Literature Mathematics Science Perf. Assessments Writing Math Problem Solving All with Adaptations Accommodations Modifications Extended Perf. Measures X-Reading X-Math X-Writing X-CLRAS Oregon’s Comprehensive Assessment System In 1999-2000 Almond, Karvonen Savannah, GA March 2006

19. NCEO Description of Accommodations “Accommodations are changes in testing materials or procedures that enable students to participate in assessments in a way that allows abilities to be assessed rather than disabilities. They are provided to ’level the playing field.’ Without accommodations, the assessment may not accurately measure the [individual] student’s knowledge and skills.” Emphasis added. Almond, Karvonen Savannah, GA March 2006

20. Accommodations and Modifications Tables in Test Administration Manuals Almond, Karvonen Savannah, GA March 2006

21. The Settlement Feb. 2001 . . . Accommodations shall be considered allowable, valid, and scorable if they are used during instruction[1] . . . and are listed on a student’s IEP or Section 504 plan, unless [the department] can show that the accommodation invalidates the score interpretation.[2] Almond, Karvonen Savannah, GA March 2006

22. The Settlement Feb. 2001 continued . . . Rather than consider all accommodations first invalid until proven to be valid, [the department] shall consider all accommodations valid unless [it] can show that the accommodation would invalidate the score interpretation. Almond, Karvonen Savannah, GA March 2006

23. The Settlement Feb. 2001 continued Any list of approved or not approved accommodations published by [the department] as a guide for school districts shall not be deemed exclusive . . . Almond, Karvonen Savannah, GA March 2006

24. The Settlement Feb. 2001 continued When . . . a particular accommodation would invalidate the score interpretation, [the department] must provide an alternate assessment and/or alternate scoring procedures . . . to afford . . . students an assessment option that leads to the Certificate of Initial Mastery. Almond, Karvonen Savannah, GA March 2006

25. 2000-2001 Expanded Menu of Testing Options for Full Participation • Regular Tests & Achievement Level Testing (Rdg/Lit & Math)* • Challenge up or down • Accommodations • Modifications • Extended Reading, Math, Writing • Ext. Career and Life Role Assessment • Juried Assessment * Regular refers to tests administered with and without accommodations. Almond, Karvonen Savannah, GA March 2006

26. Juried Assessment • Juried CIM Assessment • Juried Modifications Almond, Karvonen Savannah, GA March 2006

27. Pathways to Certificate of Initial Mastery (CIM) Collections • ODE Collection of Evidence • OR University System—Proficiency Based Admission Standards System (PASS) • Statewide Assessments • CIM level State Tests (w/ or w/o accommodations) • CIM level State Tests w/ individually approved modifications Almond, Karvonen Savannah, GA March 2006

28. Juried Modifications • Available at CIM Level • District brings request forward • Student must be able to meet state standards • Decision is student-by-student & test-by-test Almond, Karvonen Savannah, GA March 2006

29. “Three examples of such requests for a juried review of a modification follow: • A student literate in a language other than English, Spanish or Russian, wishes to take the mathematics test translated into his or her primary language. Such a translation would typically be considered a modification. 2. A student with limited physical dexterity writes using a special word processing program that creates words or phrases from a few keystrokes and the student wishes to take the writing test using the software. 3. A student with a significant learning disability such as dyslexia, affecting the student’s ability to read and write, uses auditory methods such as taped texts and readers as a primary learning mode.” Almond, Karvonen Savannah, GA March 2006

30. “The panel might approve this modification as an accommodation for the particular student after reviewing the student's case if: The student is skilled in using the read aloud adaptations The measure of comprehension reflects the student’s own knowledge and understanding The student achieves the same standards for interpreting text required of all students” “ . . . a significant learning disability who uses assistive technology, screen readers, and recorded text . . . understanding text and interpreting “meaning”. Almond, Karvonen Savannah, GA March 2006

31. “If approved, the student would be permitted to use the "read aloud" modification with the Reading/Literature Knowledge and Skills assessment and have the opportunity to "meet" (e.g. be determined "proficient" on the standard.)” Almond, Karvonen Savannah, GA March 2006

32. Accommodations in Oregon Almond, Karvonen Savannah, GA March 2006

33. The Accommodations Panel Almond, Karvonen Savannah, GA March 2006

34. Purpose: Conduct review of proposed additions to the accommodations table for the test administration manual to assure that that the adaptation will not invalidate interpretations and judgments based on the resulting test score. Almond, Karvonen Savannah, GA March 2006

35. Process • Proposal to add adaptation to the Accommodations Table • Staff analysis of relevant research and legal decisions about the effects of an adaptation on score validity • Accommodations Panel review of evidence and deliberation Almond, Karvonen Savannah, GA March 2006

36. Results • A recommendation to include the adaptation either on the Accommodations Table or the Modifications Table • A decision by the Associate Superintendent of Assessment and Evaluation about the status of the adaptation • Publication in test administration manual and revised tables Almond, Karvonen Savannah, GA March 2006

37. Deaf Educator Math Content Specialist* ESL Teacher Vision Specialist District Test Administrator Special Ed. Director English Language Arts Content Specialist* Parent Educational Measurement Expert University Researcher School Principal Special Ed. Teacher Accommodations Panel Membership * Formerly assessment item writers and assessment content panel members. Almond, Karvonen Savannah, GA March 2006

38. Panel members will: • Review adaptations, research reports, and assessment materials, • Participate in deliberations and decision-making based on review criteria, and • Advise the department regarding test accommodations and modifications Almond, Karvonen Savannah, GA March 2006

39. Member Responsibilities: • Represent area of knowledge and expertise represented by position on panel • Be familiar with the State Assessments • Attend all meetings • Apply review criteria fairly and ethically • Work collaboratively with other members Almond, Karvonen Savannah, GA March 2006

40. Panel Member Requirements: • Knowledge: Masters or analogous • Education Experience: 5 years or equivalent • Advisory Experience: Prior experience on advisory or content panel • Test Experience: Prior testing experience • Teamwork: Effectively work as a member of a group Almond, Karvonen Savannah, GA March 2006

41. Terminology for Testing Accommodations ADAPTATION • Change made to the environment, curriculum, instruction, and/or assessment practices in order for a student to be a successful learner. Adaptations include accommodations and modifications. Adaptations are based on an individual students’ strengths and needs, and may vary in intensity and degree (originally from the Colorado Department of Education, 1995). Almond, Karvonen Savannah, GA March 2006

42. Terminology continued ACCOMMODATION • An alteration in how a test is presented to or responded to by the person tested; includes a variety of alterations in presentation format, response format, setting in which the test is taken, timing or scheduling. The alterations do not substantially change level, content or performance criteria. The changes are made in order to provide a student equal access to learning and equal opportunity to demonstrate what is known. Almond, Karvonen Savannah, GA March 2006

43. Terminologycontinued MODIFICATION • Substantial changes in what a student is expected to learn and/or demonstrate. The changes are made to provide a student opportunities to participate meaningfully and productively in learning experiences and environments. They include changes in instructional level, content, and performance criteria. Almond, Karvonen Savannah, GA March 2006

44. Review Criteria • Standards • Purpose • Function • Consequence • Social Consequence • Systems Change Almond, Karvonen Savannah, GA March 2006

45. Review Criteria Continued:1. Content and Performance Standards: • What is intended to be measured by the assessment? • Clarification for this question provided by department of education staff. Almond, Karvonen Savannah, GA March 2006

46. Review Criteria Continued:2. Purpose: • What is the requestor’s intent? • For example, to enhance access to test or to increase number of students who can participate? • Why was the request made to add the adaptation? Almond, Karvonen Savannah, GA March 2006

47. Review Criteria Continued:3. Function: • What is the effect of the adaptation? • How does it work? • How will the accommodation be used? • How does it need to be done? For example, scribe accommodation required protocols for training and implementation. Almond, Karvonen Savannah, GA March 2006

48. Review Criteria Continued:4. Consequence: • What inferences are being made or jeopardized? • What mistakes will be made if the adaptation is used as an accommodation? Ripple effect? Almond, Karvonen Savannah, GA March 2006

49. Review Criteria Continued:5. Social Consequences: What affect will the testing adaptation have on students across settings? What is the impact on students of using the adaptation? Also look more globally—Does adaptation influence general perceptions of the public about the validity of assessment? Almond, Karvonen Savannah, GA March 2006

50. Review Criteria Continued:6. System change: • Would proposed adaptation constitute a system change outside the Accommodation Panel’s sphere of responsibility? • Does it propose a change to the assessment system and/or to the content and performance standards? Almond, Karvonen Savannah, GA March 2006