California's Integrated Accountability and Continuous Improvement System

1. California Practitioners Advisory Group MeetingItem 2: California’s Integrated Local, State, and Federal Accountability and Continuous Improvement System Analysis, Measurement, and Accountability Reporting Division August 2019

2. Today’s Topics • Use of Status for English Learner Progress Indicator (ELPI) in Local Educational Agency (LEA) and School Assistance Determinations • Incorporation of the California Alternate Assessments (CAAs) into the Academic Indicator • Modified Cut Scores for the Academic Indicator for Dashboard Alternative Schools Status (DASS) • Revised Cut Scores for the Graduation Rate Indicator • Revisions to District of Residence Business Rules California Department of Education

3. Use of ELPI Status in Local Control Funding Formula (LCFF) Local Educational Agency and Every Student Succeeds Act (ESSA) School Assistance Determinations California Department of Education

4. Use of Color in Determining LCFF LEA Eligibility for Assistance • To be eligible for LCFF support, LEAsmust have at least one student group that meets the criteria in more than one priority area • Generally, a student group with a ‘red’ color on two state indicators meets the criteria. • One exception is Priority Area 4 (Pupil Achievement), which includes three state indicators, namely English Language Arts/Literacy (ELA), Mathematics (Math), and ELPI.

5. California State Plan ELPI Amendment Request In December 2018, California submitted the SBE approved state plan amendment to include ELPI Status in a consecutive year of Comprehensive Support and Improvement (CSI)/Additional Targeted Support and Improvement (ATSI) determinations using the 2019 Dashboard. This is the earliest time feasible to have a valid and reliable ELPI after the transition to the ELPAC. This is because the ELPI requires two years of ELPAC Summative Assessment data to calculate Status as it is a measure of student progress.

6. Example Including ELPI Status for LCFF District Eligibility for Assistance Determination Example 1: Crystal Unified School District Performance levels achieved by EL student group in all applicable indicators:

7. Example Including ELPI Status for Eligibility in theComprehensive Support and Improvement (CSI) Example 2: Amethyst Elementary School Amethyst received Title I funds for the 2018–19 school year and would be eligible for CSI because it met the criterion of having five or more indicators where the majority are Red or the ELPI Status is “Very Low”.

8. ELPI Discussion • Do you have questions or concerns with the proposal to use ELPI Status to identify districts for LCFF support? California Department of Education

9. Incorporating the CAA into the Academic Indicator California Department of Education

10. Current Methodology for Academic Indicator • “Distance from Standard” (DFS) on the Smarter Balanced Summative Assessments (SBAC) • SBAC aligned to the California Common Core State Standards • Students who take the SBAC are placed in one offour achievement levels • Distance between student’s score on the SBAC and the “Standard Met” Achievement Level threshold (i.e., the lower threshold of the SBAC scale score range for Level 3). California Department of Education

11. DFS Example: Grade 5 Smarter Balanced Summative Assessments in Mathematics A Grade 5 student received a score of 2,505. Student scored 2,505. This is 23 points below the lowest possible score for Level 3. The student’s DFS for mathematics is -23 points. (2,505–2,528 = -23 points)

12. Incorporating CAA Results into Academic Indicator • Every Student Succeeds Act (ESSA) requires states to include assessment results of all students—including those with significant cognitive disabilities—into accountability systems. • CAA is already included in participation rate calculations but excluded from calculation of DFS (which is used to determine performance colors). • Beginning with the 2019 Dashboard, the CAA results will be included in the calculations for the Academic Indicator. California Department of Education

13. Challenges for Incorporating CAA Results into Academic Indicator • CAA based on different set of standards than those used for SBAC: • Common Core Alternate Standards • Students evaluated against their level of “understanding” (rather than meeting the standard, as in the case for the SBAC). • SBAC and CAA have different reporting scales with distinct distributions. California Department of Education

14. Additional Challenges for Incorporating CAA Results into Academic Indicator • Students taking the CAA are placed in one of three levels of understanding (in contrast to one of four levels for the SBAC) • Level 1 is “limited understanding” • Level 2 is “foundational understanding”’ • Level 3 is “understanding” of alternate standards • Sample sizes vary between the two assessments California Department of Education

15. Developing a Methodology for Incorporating the CAA Results into the Academic Indicator • TDG has explored three methodologies over the course of three meetings: • Effect Size • Middle of the Scale Range • Top of the Scale Range California Department of Education

16. Effect Size Methodology • Apply the DFS methodology to the CAA, where “Standard” represents “understanding of alternate standards” (Level 3). • Convert CAA scores using the standard deviations for each of the two assessments. • For example, if a CAA score is 0.5 standard deviations from Level 3 that CAA score is converted to the SBAC score that is 0.5 standard deviations from Level 3, as shown on next slide. California Department of Education

17. Example of Effect Size Grade 3 English Language Arts Student 0.5 Standard Deviation Above Level 3 Threshold Score on CAA (370) Translated to 0.5 Standard Deviation Above Level 3 Threshold Score on SBAC (2,477) California Department of Education

18. Middle-of-the-Range Methodology • For performance levels 1–3 on the CAA, a student’s CAA score would be substituted with the mid-range score of the same SBAC achievement level. • Example: A grade 3 student scoring anywhere in Level 2 on CAA for ELA would receive the midpoint score of the Level 2 range on the SBAC ELA, 2399 (SBAC scale range is 2367-2431). California Department of Education

19. Top-of-the-Range Methodology • For performance levels 1–3 on the CAA, a student’s score would be substituted with the top score of the same SBAC achievement level. • Example: A grade 3 student scoring anywhere in Level 2 on CAA for ELA would receive the highest score of the Level 2 range on the SBAC ELA, 2431. California Department of Education

20. Statewide Impact of Effect Size on English Language Arts Note: The Lowest Obtainable Scale Score (LOSS) were excluded

21. Statewide Impact of Middle Scale Range onEnglish Language Arts Note: Note: LOSS scores were excluded

22. Statewide Impact of Top Scale Range on English Language Arts Note: LOSS scores were excluded

23. Impact Summary for the Three Methodologies • Removing students who received a LOSS improved outcomes. • The DFS for students with disabilities (SWDs) improved, under all three methodologies, when the CAA was incorporated. This is because the scores of students who take the CAA are generally closer to Level 3 (for that test) than are the scores for SWDs who take the SBAC (i.e., DFS is generally greater). • Top of the scale range methodology, excluding LOSS scores, resulted in the smallest difference between DFS scores for the SBAC and CAA. California Department of Education

24. CDE Recommendation • Top-of-the-Range with the exclusion of the CAA LOSS scores • Easier to communicate than the Effect Size • Will be proposed to the State Board of Education (SBE) in September 2019. California Department of Education

25. Why Top of Range vs. Middle of Range? • Both methodologies limit the range of CAA conversions to three SBAC performance level (1-3). • Only the Effect Size allows CAA scores to be converted across all four SBAC performance level • Middle of Range further limits the range of SBAC scores available for CAA conversion, while the Top of Range takes full advantage of three complete ranges of SBAC scores. California Department of Education

26. CAA Questions/Discussion • Do you have any concerns with the recommended methodology of Top of Scale Range? If so, what are your concerns? • Would you prefer a different methodology? If so, which methodology and why?

27. Modified Cut Scores for the Academic Indicator California Department of Education

28. Modified Measures for DASS Schools • DASS schools are held accountable for meeting all of the same state indicators that are currently reported in the Dashboard. • However, in order to fairly evaluate the success of alternative schools that serve high-risk students, modified measures have been developed for specific indicators (e.g., grade twelve graduation rate and collection of career measures specific to DASS schools for the College/Career Indicator). • The CDE now proposes a modified set of Status cut scores for the Academic Indicator. California Department of Education

29. No New Revisions Proposed for Change Cut Scores • Change distributions for DASS schools are not markedly different than the current LEA distributions. • In addition, the CDE believes that maintaining the same Change cut scores for DASS schools reflects the expectations for continuous improvement model under California’s accountability system. California Department of Education

30. Comparing LEA and DASS Distributions for Status • Significant differences at the 50thpercentile (Handout 1). • Majority of DASS schools have a negative DFS • CDE recommends that changes to the DASS cut scores be limited to the Low and Very Low Status levels. • Will be proposed to the SBE in September 2019 • Proposed DASS cut scores appear in Tables 1 through 4 California Department of Education

31. Table 1: Proposed DASS Cut Scores for ELA, Grades Three through Eight 12 DASS schools move from “Very Low” to “Low” Status level Number of DASS schools receiving “Very Low” Status level decreases from 31 to 19 Number of DASS schools receiving “Low” Status level increases from 9 to 21

32. Table 2: Proposed DASS Cut Scores for ELA, Grade Eleven 56 DASS schools move from “Very Low” to “Low” Status level Number of DASS schools receiving “Very Low” Status level decreases from 141 to 85 Number of DASS schools receiving “Low” Status level increases from 12 to 68

33. Table 3: Proposed DASS Cut Scores for Mathematics, Grades Three through Eight 19 DASS schools move from “Very Low” to “Low” Status level Number of DASS schools receiving “Very Low” Status level decreases from 39 to 20 Number of DASS schools receiving “Low” Status level increases from 5 to 24

34. Table 4: Proposed DASS Cut Scores for Mathematics, Grade Eleven 45 DASS schools move from “Very Low” to “Low” Status level Number of DASS schools receiving “Very Low” Status level decreases from 148 to 103 Number of DASS schools receiving “Low” Status level increases from 4 to 49

35. DASS Academic Indicator: Questions/Discussion • Do you have any questions or concerns with the proposed Academic Indicator cut scores for DASS schools? California Department of Education

36. Revised Cut Scores for the Graduation Rate Indicator

37. Adoption of Combined Four- and Five-Year Graduation Rate • At its July 2019 meeting, the SBE adopted a combined four- and five-year graduation rate for the Graduation Rate Indicator. • Since new the methodology increases graduation rates, the SBE determined that the State should raise the Graduation Rate “threshold” to determine eligibility of schools for Comprehensive Support and Improvement (CSI).

38. CSI Eligibility Criteria • Low Graduation Rate: Based on a two–year average graduation rate • Applies to both Title I–funded and non-Title I-funded schools • Lowest Performing: Only for schools that received Title I funding • At least five percent of the Title I–funded schools must be identified in this category

39. CSI Process • Since a school can only be identified in one category for any given school year, the first category takes precedence. • Schools with low graduation rates are identified first. • The lowest performing schools, which must comprise at least five percent of all Title I-funded schools, are identified from the remaining pool. California Department of Education

40. Resetting Threshold for Low Graduation Rate • Currently set at 67 percent • Schools with graduation rates below 67 percent are identified for CSI • Two new thresholds—68 percent and 70 percent—proposed for consideration • Adopting a new graduation rate threshold for CSI identification results in new Status cut scores for the Graduation Rate Indicator

41. Impact of New Threshold • Raising the threshold for the “low graduation rate” criterion for CSI could reduce the number of schools identified under the lowest performing criterion. • To determine the impact, and ensure that California continues to meet the ESSA requirement of identifying at least five percent of the lowest performing Title I schools, the CDE conducted several simulations.   California Department of Education

42. Simulations Conducted • Using 2018 Dashboard grade 12 graduation rates, simulations were conducted to identify the number of DASS high schools that would be eligible for CSI if the threshold “for low graduation rate” were raised. • Using the 2017–18 combined graduation rate, simulations were conducted to identify the number of non-alternative high schools that would be eligible for CSI if the threshold for “low graduation rate” were raised.   California Department of Education

43. Simulation Results: Threshold Raised to “Below 68 Percent” • 41 additional schools identified for CSI under “Low Graduation” rate • 16 DASS Schools • 25 Non-DASS Schools • Of the 41 schools, only two schools already identified as part of the five percent of lowest-performing Title I schools California Department of Education

44. Simulation Results: Threshold Raised to “Below 70 Percent” • 62 additional schools identified for CSI under “Low Graduation” rate • 30 DASS Schools • 32 Non-DASS Schools • Of the 62 schools, 8 schools already identified as part of the five percent of lowest-performing Title I schools California Department of Education

45. Impact of Revised Cut Scores • This change will result in new Status cut scores for both the non-DASS schools and DASS schools   • If the threshold is set at 70%, DASS schools will no longer have a separate five-by-five colored table (see next slide) California Department of Education

46. Colored Tables Non-DASS DASS California Department of Education

47. Graduation Threshold: Questions/Discussion • Do you have a preference on the low graduation rate threshold (less than 68% or less than 70%)? • If so, what is your preference and why? California Department of Education

48. Revisions to the District of Residence/ Attendance Business Rules California Department of Education

49. Monitoring under IDEA The primary focus of the State's monitoring activities must be on: • Improving educational results and functional outcomes for all children with disabilities; and • Ensuring that public agencies meet the program requirements under Part B of the Act, with a particular emphasis on those requirements that are most closely related to improving educational results for children with disabilities. California Department of Education

50. Monitoring Priority Areas under IDEA The State must monitor the LEAs located in the State, using quantifiable indicators in each of the following priority areas: • Provision of a free appropriate public education (FAPE) in the least restrictive environment. • State exercise of general supervision, including child find, effective monitoring, the use of resolution meetings, mediation, and a system of transition services. • Disproportionate representation of racial and ethnic groups in special education and related services, to the extent the representation is the result of inappropriate identification. California Department of Education