Exploring Alternate AYP Designs for Assessment and Accountability Systems

1. Exploring Alternate AYP Designs for Assessment and Accountability Systems Dr. J.P. Beaudoin, CEO, Research in Action, Inc. Dr. Patricia Abeyta, Bureau of Indian Education NIEA 2008 Annual Conference Seattle, Washington

2. Session Goals • Frame the technical and resource considerations for decision-makers • Provide clarification on the regulatory provisions associated with NCLB • Share assessment and accountability design features

3. Guiding Questions P • urpose • How does the purpose for the change/action serve Native children? • irection • What direction will the change/action provide the educational community? • otivation • To what degree will the change/action motivate the Native peoples? D M

4. 30,000ft. SYSTEM VIEW Native Heritage

5. School System Components • Academic Content Standards • “What do students need to know and be able to demonstrate?” • Instruction • “How is content organized, provided, modeled, and expanded upon?” • Assessment • “To what degree is the content expressed accurately by the leaner?” • Accountability • “Which learning indicators are being reached, showing progress, or remaining unchanged?”

6. Integration • Things to Consider… • Changes to one component will have some influence on the remaining components • Improvements in one area often result in benefits throughout the system • Isolated changes create bulkization, inefficiencies, and internal stressors

7. Leverage

8. Compliance If you change the academic content standards used for AYP… If you change the assessments used for AYP… If you change how AYP is defined… Then you must provide evidence that addresses Section 1 of the USDE’s Peer Review Guidance Then you must provide evidence that addresses Sections 2-7 of the USDE’s Peer Review Guidance Then you must create an AYP Workbook that address 10 accountability principles.

9. Overview: Standards and Assessments Peer Review Conditioned on what is changed, evidence must be presented to the BIE and USDE Peer Reviewers… Section 1-Content Standards Section 2-Achievement Standards Section 3-Coherent Assessment System Section 4-Technical Quality Section 5-Alignment Section 6-Inclusion Section 7-Reporting

10. Overview: AYP Workbook Conditioned on what is changed, policy and empirical evidence must be submitted for review by the BIE and USDE… Principle 1-Includes all schools and districts Principle 2-Includes all students Principle 3-Includes an allowable method for AYP determinations Principle 4-Includes annual accountability decisions Principle 5-Includes subgroup accountability Principle 6-Based on academic assessments Principle 7-Has additional indicators Principle 8-Separate decision for reading and mathematics Principle 9-System validity and reliability Principle 10-Includes participation rates

11. Overview: Review Process

12. ASSESSMENT

13. Generic Assessment Life Cycle

14. Foundational Components • Academic content standards • Articulate what student are expected to know and be able to do • Grade level expectations in reading/ELA and mathematics • Grade span clusters in science • Contain skill and process knowledge

15. Foundational Components • Academic achievement standards • Performance level descriptors • No less than three achievement levels (with applicable labels) • “Cut-Score” established using a technically valid standard-setting approach (e.g, Modified Bookmark, Body of Work, Teacher Judgment) • Established by subject matter experts (teachers) and other panelists with applicable skills

16. Foundational Components • Assessments aligned to intended purposes • Assessments in grades 3-8 and high school: • Reading/ELA • Mathematics • Science (3 grade clusters) • Standardized administration and scoring procedures

17. Foundational Process • Typically 18-24 month development process • Documented evidence • Reviewed by external (to USDE) experts • Evaluated evidence against Peer Review Guidance • Resubmissions typical

18. USDE PEER REVIEW GUIDANCEStandards and Assessments

19. Section 1: Academic Content Standards Students know and are able to… • Content standards are: • Adopted in mathematics; reading/ELA • Adopted in science (grade spans) • Coherent and rigorous • Developed by stakeholders

20. Section 2: Academic Achievement Standards The level students have mastered the content is… • Achievement standards are: • Adopted in mathematics, reading/ELA, and science • Assigned achievement levels and content-based descriptors • Established using an appropriate standard-setting procedure • Aligned to the content standards

21. Section 3: Coherent Assessment System The different assessment components within the larger assessment system contribute… A coherent system: • Is based on state and/or local assessments • Provides integrated achievement data • Aggregates only comparable measures • Utilizes multiple measures focused on higher order thinking skills • Implements alternate assessments

22. Section 4: Technical Quality The technical quality of the assessments is… High quality assessments are: • Valid measures based on their purpose • Constructed to measure the content standards • Accurate and consistent • Fair and accessible • Consistent across different forms and events • Standardized in their administration and scoring

23. Section 5: Alignment The assessments reflect the content standards characteristics by… Strong alignment is provided by using: • Using procedures to improve alignment • Techniques to match the content standards and patterns of emphasis • Capture the range and depth of knowledge in the standards • Pattern of emphasis • Approaches to measure both content and process knowledge • Content-based performance descriptors

24. Section 6: Inclusion The assessment system includes all students by… Students participate in the assessments as demonstrate by: • Impact data showing high rates of student participation • Affording students with disabilities participation under accommodated conditions • Accommodating students learning to speak English • Including migrant and other highly mobile students

25. Section 7: Reporting The assessment reports provide accurate and timely information about student achievement by… Student achievement data is: • Reported using valid and credible procedures • Based on all students from valid assessments • Reported at the individual student, subgroup, school, district, and state-levels • Disseminated in a timely manner • Easily understood by parents and the public

26. USDE REVIEW GUIDANCEAYP Workbook

27. Principal 1: Includes All Schools and Districts • Includes all schools and districts • Holds all schools to the same criteria • Incorporates the academic achievement standards • Provides information in a timely manner • Includes report cards • Includes rewards and sanctions

28. Principal 2: Includes All Students • Includes all students • Consistently defines full academic year (FAY) • Includes mobile students

29. Principal 3: Method of AYP Determinations • Applies universal proficiency by 2013-2014 as the long-term goal • Uses a method to make subgroup, school, and district AYP determinations • Establishes a starting point with statewide annual measurable objectives (AMOs) • Establishes intermediate goals

30. Principal 4: Includes Annual Decisions • Determines annually the progress of schools and districts

31. Principal 5: Includes Subgroup Accountability • Includes all the required student subgroups • Holds schools and LEAs accountable for the progress of student subgroups • Includes students with disabilities (SWD) and limited English proficient (LEP) students • Includes an established minimum n-count • Includes safeguards to protect student privacy

32. Principal 6: Based on Academic Assessments • Based primarily on academic assessments

33. Principal 7: Includes Additional Indicators • Includes graduate rate for high schools • Includes an additional academic indicators for elementary and middle schools • Establishes valid and reliable additional indicators

34. Principal 8: Includes Separated Decisions for Reading/ELA and Math • Holds students, schools, and districts separately accountable for reading/ELA and mathematics

35. Principal 9: System Validity and Reliability • Produces reliable decisions • Produces valid decisions • Addresses changes in assessments and student populations

36. Principal 10: Includes Participation Rates • Includes a way to calculate the rate of participation on the statewide assessments • Applies the 95% participation criteria to student subgroups and small schools

37. Research in Action, Inc. World-Class Educational Quality™ Voice: 1.225.571.2408 E-Mail: ria2001@eatel.net Fax: 1.225.644.8472 Website: www.ria2001.org Research in Action, Inc.