Considerations for the Alternate Assessment based on Modified Achievement Standards

1. Considerations for the Alternate Assessment based on Modified Achievement Standards Briefing on a white paper commissioned by the New York Comprehensive Center in collaboration with the New York State Education Department January 11, 2010 The contents of this publication were developed under cooperative agreement S283B050019 with the U. S. Department of Education. However, the contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.

2. Introductions and Project Overview • NYCC received a letter from USED informing them that funds were available for a research project • NYSED agreed that an expert panel meeting culminating in a white paper on issues surrounding the AA-MAS would be useful • NYCC applied for the funds and contracted with NCIEA to support this work

3. Issues Facing NY and Most States • New regulation without much discussion with states led to many questions • Should states spend limited resources to develop an AA-MAS? • Will it yield useful information to guide instruction? • Is that the best way to support the learning of this population? • Would this test only be helpful to schools for accountability purposes? • Are there current best practices with this population in assessment development that should be followed? • What does the research say?

4. NYSED Initial Questions Regarding the AA-MAS • Which students are best served by this assessment? • How different are they from the rest of the special education population? • What is an “appropriately challenging” achievement standard? • Which modifications make the most sense in the context of the AA-MAS? • How do the modifications affect the validity and reliability of the interpretation? • What is the credential that is most appropriate for students participating in the AA-MAS and what does it lead to in terms of post-secondary potential?

5. Overview of Project • NYCC contracted with the Center for Assessment to assemble an expert panel and develop a white paper on issues concerning the development of an AA-MAS • NYSED provided guidance on important policy questions and recommended addressing issues in Filbin (2008) report • Expert panel met twice in person and several times over WebEx • Developed a 10-chapter report exploring many issues surrounding the conceptualization and development of an AA-MAS

6. Project Director: Larry Hirsch, New York Comprehensive Center Project Manager/Editor: Marianne Perie, Center for Assessment New York State Department of Education: David Abrams, Assistant Commissioner for Standards, Assessment, and Reporting Candy Shyer, Bureau Chief of Test Development, Office of State Assessment Rebecca Cort, Deputy Commissioner, Office of Vocational and Educational Services for Students with Disabilities Expert Panel: Jamal Abedi, University of California, Davis Chris Domaleski, Center for Assessment Steve Dunbar, University of Iowa Howard Everson, Fordham University Claudia Flowers, University of North Carolina, Charlotte Brian Gong, Center for Assessment Meagan Karvonen, Western Carolina University Suzanne Lane, University of Pittsburgh Scott Marion, Center for Assessment Jim Pellegrino, University of Illinois, Chicago Marianne Perie, Center for Assessment David Pugalee, University of North Carolina, Charlotte Rachel Quenemoen, National Center on Educational Outcomes Robert Rickelman, University of North Carolina, Charlotte Katherine Ryan, University of Illinois, Urbana Champagne Gerald Tindal, University of Oregon Cathy Welch, University of Iowa Phoebe Winter, Pacific Metrics Project Participants

7. Authors and Reviewers • NYSED and the Center for Assessment identified chapter authors and reviewers and sent requests prior to the first expert panel meeting • Eleven expert panelists served as authors • The remaining seven experts served as reviewers • Discussions were organized by section & each reviewer was assigned a section • Three reviewers read all chapters

8. Organizing Structure of Report • Report is organized into three sections • Identifying and understanding the population • Test Design: Understanding content and achievement standards and incorporating appropriate item modifications • Technical considerations and practical applications • Or… • Should I develop an AA-MAS? • How should I develop it? • How should I evaluate it?

9. Organization (continued) • 10 chapters within the three sections • Introduction Identifying & Understanding the Population • Identifying the Population • Standards-based IEPs • Cognition of Low Achievers Test Design • Reading and Math Content • Test Development • Standard Setting Technical Considerations • Comparability • Validity • Accountability and operationalization

10. Section I: Identifying and Understanding the Population • Intro and first chapter provide a policy context for the AA-MAS and a historical perspective on educating students with disabilities • Includes a discussion framework for state policymakers on assessment options and on improving student access to grade-level curriculum

11. Section I (continued) • Second chapter describes how a standards-based IEP supports an educational program that meets the needs of AA-MAS-eligible students, through • access to grade-level curriculum using effective instruction; • supports to address learner characteristics; and • setting and monitoring goals to support a transition into grade-level achievement. • Third chapter describes relationships among assessment, curriculum, and instruction, and conceptualizing assessment as a process of reasoning from evidence driven by theories and data on student cognition, and includes examples related to the AA-MAS.

12. Section II: Test Design • Starts with a chapter describing content domains for ELA and Math with a focus on ways to modify content so that we can measure greater depth with easier items • Next chapter discusses item and test development and reviews approaches to modification of items and assembly of test forms with respect to psychometric consequences and standards-based interpretations of proficiency for the AA-MAS population.

13. Section II (continued) • Final chapter in this section focuses on developing modified achievement standards • Includes a discussion on how to define “modified” proficiency, synthesizing information from the previous sections on cognition and test design, and provides ideas for setting cut scores on assessments with fewer items and lower sample sizes

14. Section III: Technical Considerations • First chapter in this section discusses comparability between AA-MAS and general assessments from different perspectives including • content and construct, • psychometrics, • scale and score, • linguistic structure, • text features, and • depth of knowledge. • The next chapter describes why and how to develop a validity argument using Kane’s argument-based approach as a framework for considering validity issues related to AA-MAS.

15. Section III (continued) • The final chapter in this section describes operational and accountability issues and is specifically geared towards policymakers • It focuses on the interrelationship of the AA-MAS to the existing state assessment and accountability system considering several practical, technical, and policy issues that must be considered when implementing a new program.

16. Other Features • Other resources developed at the request of NYSED • List of additional web-based resources on instructing and assessing students with disabilities • Tool for policymakers that includes guiding questions for state policymakers considering the development of an AA-MAS and references back to specific sections within the report • Glossary of terms related to special education, assessment, and federal policy

17. Overarching Themes • Clear link between assessment, instruction, and student cognition, • Several of the chapters focus on the importance of bringing lessons learned in studying the students and designing the assessments into the classroom. • Similar to accommodations, certain modifications will only be successful to the degree they are incorporated into a student’s daily instruction. • Important to develop a validity argument for this assessment early in the process and test the various assumptions throughout. • Testing the assumption that a new assessment is needed is one of the first important recommendations. • Many chapters address ongoing collection and evaluation of validity evidence to ensure that the development is in line with the expected goals. • Consider how to incorporate the recommendations into an existing assessment and accountability system • Including how to work with current state content standards and grade-level achievement standards.

18. Suggested Uses • State policymakers should find this report useful in determining whether or not developing an AA-MAS makes sense for their state, and if so, how. • Even in states with no intention of designing an AA-MAS, this report could be a useful tool for designing professional development activities. • Contains suggestions for several areas: special education, curriculum & instruction, assessment, and accountability

19. Lingering Concerns • Still concerned about identifying right population • Kids are classified differently from one school to the next • We need to understand the need for such classifications and find ways to better classify them into instructionally useful categories • If AA-MAS modifications work well for these kids, why not expand the approach for all assessments • Need more focus on universal design, better accommodations, and aligning assessments with diagnostic approaches to learning.

20. Lessons Learned from the Report • This report is applicable to multiple tests and types of students – not just the “2%” population. • We need to consider how the AA-MAS fits in with the general assessment. • It highlighted the importance of an integrated test design, meaning we must consider assessment, curriculum, and instruction together in consideration of how tests will be used and validated.

21. Bigger Lessons Learned • Key decision points should be identified and informed by research and best practices. • There is a value in triangulating federally-funded research with states’ operational work: • States need time to have thoughtful discussion to process issues. • This type of work increases professional development of state department staff. • It helps inform several areas like RTTT and allows measurement research to inform policy.

22. Lessons Learned on Next Steps • We need to strengthen our understanding of learning progressions and integrate that understanding into classroom instruction and assessment development. • Where do students with disabilities differ from general population? • Need to understand more than scope and sequence • Should provide information on next steps to help strengthen nuances of student learning

23. Next Steps for New York • Will not develop an AA-MAS under current political context • Need to see what happens with ESEA reauthorization and RTTT • In the meantime… • This work is helping us think through formative and interim assessment strategies. • We are also working with our colleagues in special education to rethink IEP design.