In Search of Consistency: Guidelines for the Administration of Accommodations by Third Parties

1. In Search of Consistency: Guidelines for the Administration of Accommodations by Third Parties

2. Presenters Ann Clapper, National Center on Educational Outcomes Amanda Blount Morse, National Center on Educational Outcomes Janet Filbin, Colorado Department of Education Scott Trimble, Kentucky Department of Education

3. What are third parties? Why are guidelines important? What is the current status of state policies on third parties? How are states addressing the issue of third parties? What are key elements of the process and the product? Session Overview

4. Reader Reads test directions, items, passages, and/or prompts orally to students who are unable to decode text visually. Scribe Writes down what a student dictates through speech, sign language, pointing or by using an assistive communication device. Sign Language Interpreter Translates test directions and/or items from spoken English into American Sign Language or other method of sign language. The sign language interpreter also allows students to respond to written questions in sign language. Who are we talking about?

5. Standardization is an essential feature of educational assessments designed to support comparisons among participants. Strict adherence to guidelines detailing instructions and procedures for the administration of accommodations is necessary to ensure that the test results reflect actual student learning. If accommodations are administered inconsistently the results will not be comparable across examinees. The Importance of Guidelines

6. The Number of States With Existing Guidelines

7. COLORADO Student Assessment Program CSAP Janet Filbin Colorado Department of Education Filbin_j@cde.state.co.us (303) 866-6703

8. The Colorado Student Assessment Program (CSAP) • Legislated in 1993 through HB 1313 • Criterion-referenced assessment originally designed to obtain a point in time picture of 3rd -4th -7th and 10th graders in the area of reading and writing 5th -8th -10th grade students in math 8th grade students in science • Provides state, district and building level information about how students from year to year were progressing toward state standards • Requires all students have opportunity to learn state standards

9. CSAP Accommodations Accommodationsare changes made to the assessment procedures in order to provide a student with access to information and an equal opportunity to demonstrate knowledge and skills without affecting the reliability or validity of the assessment.

10. Eligibility for CSAP Accommodations • Any student may receive an accommodation that is allowed on the assessment if that student has received the same accommodation during daily classroom instruction for at least three months prior to the assessment. • Only accommodations the student requires for the specific content area may be selected. • The possible effects on the student of allowing the accommodations must be considered. • A student may be allowed more than one accommodation. • A student who is an English Language Learner is entitled to the same instructional and assessment accommodations as his or her English-speaking peers.

11. Development of CSAP Accommodations • Support from NCEO • Analysis of existing large scale assessment accommodations • Special Populations Advisory • Cross CDE Representatives • District Superintendents • District Assessment Coordinators • District Director’s or Specialized Services • Parent Centers

13. extended time large print Braille extended time large print Braille use of number line use of manipulatives Standard CSAP Accommodations Reading and Writing Mathematics • reading/signing directions in the student’s native language • use of scribe • signing/pointing • use of augmentative device • reading of the writing prompts only • reading/signing directions, word problems, designated answers to problems, and glossary words and definitions in the student’s native language • use of scribe • signing/pointing • use of augmentative device • oral presentation or word problems

14. Early Emerging Issues • Discrepancies in the use of accommodations from grade level to grade level, district to district and across populations • Variability in the approaches to administering accommodations • Assumed “creative license” in applying non-standard, non-approved modification to testing situations • Focus on compensatory skill development in the classroom for many students

15. Charge • Ensure access of all populations • Lack of previous large scale assessment history for ELL/special education • Reduce the potential for misadministration • Discrepancies between buildings/districts in use of accommodations • Increase clarification about the nature of the accommodation, potential users, and administration guidelines • Ongoing issue of accommodations vs. modifications • Confusion regarding state vs. district assessment accommodations • Low teacher expectations • Provide direction to districts regarding proctor qualifications • Certification/training of translators • Issues with signing

17. Stakeholders • Cross Unit Collaboration • Assessment • Title 1 • English Language Acquisition • Special Education • Association of Colorado Education Evaluators (ACEE) • CSAP Accommodations Expert Panel Review • District-level Administrators • Title I, ELL, Content, Special Education Teachers • Building Principals • CDE personnel • Parents • CSAP Bias/Content Review Committee

18. Dynamic Process Analysis of data • student performance • aggregated • special populations • participation rates • use of accommodations • special populations • by district/school • interviews with educators and administrators in the field Review of existing literature and research on accommodations (Tindal (1996); Tindal and Fuchs (1999);NCEO Studies-Thurlow, et.al. (1999 2000, 2001); Thompson, et.al. (2002), NAEP Validity Studies-Weston (2002), etc.) Examination of developed training materials

19. Recommendations • Increased guidance around specific accommodations • Scribe • Oral/signed/translated presentation • Technology • Relevant training/materials • Standard large print test • Non-standard accommodation application process • Language simplification

20. Implementation Process • Statewide CSAP Administration Training • Procedures Manual • Tools for Increasing the Performance of Students/CSAP Accommodations Video • Fast Facts • Website • Guidelines For Documenting Student Need/Teacher or Proctor Training • Considerations for Making Decisions

21. 4th Grade Reading 10th Grade Reading

22. Changes In The Use Of Reading Accommodations Over Time % of students

23. Lessons Learned • What you don’t know can hurt you • The road to standardization should not be less traveled • Just when you think your done, your not • The TAO of inservice: Once trained may not retain

24. Colorado’s Accommodation Guidelines and Training Information Homepage for the Colorado Department of Education • http://www.cde.state.co.us-homepage Procedures manual http://www.cde.state.co.us/cdeassess/csap/2004/proc_man.pdf Accommodations Video/training materials http://www.cde.state.co.us/cdeassess/csap/acc/index.htm

25. Two Basic Principles • All students can learn at high levels • Accountability must be inclusive of all students, including those at risk of failure.

26. Definethird-party providers of accommodations(who can provide the services) • Third-party providers of accommodations are trained by local districts, are either employees of or volunteers in service to the local district, and are not students. • These are typically the same persons who provide accommodations during the normal delivery of instruction.

27. Participation in Assessment Students with disabilities participate in state- required assessments: • With no accommodations, • With accommodations consistent with delivery of instruction, or • In the Alternate Portfolio Assessment.

28. Accountability Environment Kentucky’s accountability environment is considered: • High stakes for schools. • Not high stakes for students.

29. Why Accommodations? • Kentucky’s program is intended to evaluate the effectiveness of the instructional delivery system. • Students with legally identified disabilities through the ARC may have been provided special accommodations in order to access the general curriculum. • In such cases, it is critical to assess such students consistent with the instructional accommodations provided for in the IEP.

30. Questions • Why is it important to have guidelines for third parties? • What prompted your state to write guidelines for third parties?

34. Process Describe the process that Kentucky followed to develop the guidelines for third party providers of accommodations: • stakeholders • collaboration • research?

35. Questions • How was this implemented? • How is it working? • Suggestions?

36. Reviewed state guidelines Identified key elements (process and product) Drafted composite list of considerations Sent draft to 4 states and conducted telephone interviews Revised draft Development Process - Current

37. Stake-holder involvement Collaborative effort Continually evolving process Key Features - Process

38. Comprehensive Aligned Clear Specific Coherent Accessible Key Features - Product

39. The need for guidelines (i.e., access, test validity, standardization) Discussion and clarification of relevant terms (e.g., scribe, reader, etc.) Qualifications and characteristics of “third party administrators” (e.g., certified staff members, be familiar with the student, can’t be the parent of the student receiving the accommodation, have completed training) Detailed directions and procedures for implementing specific accommodations (read aloud, dictation to scribe, sign language interpretation) during testing An explanation of inappropriate administration practices Implications for test security/confidentiality Major Components

40. Are there guidelines for when and where third party test administrators can review test materials? Are there procedures for third party administrators to follow if they need clarification of test items or materials prior to test day? Are students trained on the use of a scribe, reader, and sign language interpreter during testing? Are there guidelines specific to the type of question? Are there guidelines specific to content areas? Are procedures defined for addressing third-parties’ questions or concerns that arise during testing? Specific Considerations

41. Are there requirements for how familiar the person providing the accommodation must be with the student? Are there suggestions to guide the pre-discussion between test-taker and third-party administrator? Are provisions for the settings in which accommodations can be administered described? Are there procedures that need to be followed in order to ensure that all test administrators followed the guidelines established to protect standardization and the validity of the test? Specific Considerations

42. Allman, C.B. (2003). Making tests accessible for students with visual impairments: A guide for test publishers, test developers, and state assessment personnel. Louisville, KY: American Printing House for the Blind. Educational Testing Service. (2004). ETS guidelines for a test reader. Retrieved June. 1, 2004, from ETS Web site: http://www.ets.org/disability/readguid.html Educational Testing Service. (2004). ETS guidelines for a test writer recorder. Retrieved June. 1, 2004, from ETS Web site: http://www.ets.org/disability/writguid.html Resources

43. Conduct more interviews Revise and disseminate for additional comment Write brief and develop tool Development Process - Planned

44. If you have comments or questions please contact: Amanda Blount Morse bloun005@umn.edu Ann Clapper clapp013@umn.edu Contact Information

45. National Center On Educational Outcomes Visit: http://www.nceo.info or Search for NCEO