Massachusetts’ District & School Accountability System

1. Massachusetts’ District & School Accountability System Summer/Fall 2019

2. System highlights Changes to the system for 2019 reporting Accountability indicators Normative component Criterion-referenced component Assessment participation Categorization of schools Categorization of districts Reporting Resources Agenda TOPICS

3. System highlights 1

4. Explaining “accountability” An accountability system brings together a set of measures in order to provide clear, actionable information about district & school performance. Massachusetts’ system helps schools improve the performance of all students, & helps communities & the state decide how to assign resources. Accountability results answer two questions: How is the school doing? & What kind of support does the school need?

5. System highlights • Additional accountability indicators • Provide information about school performance & student opportunities beyond test scores • Normative & criterion-referenced components • Accountability percentiles & progress toward targets • Focus on raising the performance of each school's lowest performing students • In addition to the performance of the school as a whole • Discontinuation of accountability & assistance levels 1-5 • Replaced with accountability categories that define the progress that schools are making & the type of support they may receive from the Department • Districts classified based on district-level data • No longer based on the performance of a district’s lowest performing school

6. Changes to the system for 2019 reporting 2

7. Changes to accountability system for 2019 • Expanded advanced coursework list • Includes Project Lead the Way (PLTW) • Participation rate calculation • Subgroup participation rates calculated for all subjects combined. “All students” rate remains calculated separately by subject • Using two years of data • Accountability percentiles & criterion-referenced target percentages include data from 2018 (40%) & 2019 (60%) • Progress toward improvement targets reported in four categories • Meeting or exceeding targets, substantial progress toward targets, moderate progress towardtargets, & limited or no progress toward targets

8. Accountability indicators 3

9. Massachusetts’ accountability indicators – non-high schools

10. Massachusetts’ accountability indicators – high schools

11. Grade 10 accountability reporting in 2019 • Grade 10 Next Generation MCAS results are linked to the legacy grade 10 MCAS results • Similar to process used to link PARCC results to MCAS results in 2015 (“equipercentile linking”) • Allows DESE to use the 2019 grade 10 achievement results in the accountability system this year

12. English language proficiency indicator • Set students on a non-linear path to achieving English language proficiency in six years • Set targets for each English learner based on: • Starting point (initial ACCESS for ELLs assessment results); • Grade; & • Years in Massachusetts • School & district performance is measured based on the percentage of students meeting their targets each year

13. Considerations for weighting achievement & growth • The current ratio of achievement & growth is 3 (achievement) to 1 (growth) • All indicators need to be included in the weighting • Progress towards English language proficiency only applies to a subset of schools & weighting needs to be flexible • Ratio between achievement & growth can be held constant between non-high schools & high schools but actual weightings will differ • The same weighting rules should be applied to both the normative & criterion-referenced components of the system

14. Weighting of indicators in non-high schools

15. Weighting of indicators in high schools

16. Normative component 4

17. Normative component – rationale • Federal law (ESSA) • Requires states to identify the lowest performing 5 percent of schools as needing comprehensive support & improvement • Requires states to identify schools with low performing subgroups as needing targeted support & improvement • State law • Requires that a school must be among the lowest performing 20 percent of schools statewide in order to be eligible for designation as “underperforming”

18. Normative component • Accountability percentile 1-99, calculated using all available indicators for a school • Compares schools administering similar statewide assessments • Used to identify the lowest performing schools in the state • Same calculation used at the subgroup level to identify low-performing subgroups

19. Comparisons Schools are grouped & compared based on the grades served in 2019:

20. Normative comparison groups • Most measures can be reliably compared across gradespan groupings described on previous slide • Exception is chronic absenteeism, where differences exist between school type – most notably in middle schools as compared to elementary schools • DESE will use “school type” comparison (ES, ESMS, MS, MSHS, HS) for chronic absenteeism & use that figure in overall school percentile calculation • Requires recalculating 2018 chronic absenteeism percentile

21. Comparing percentiles • 2018 accountability percentiles should not be compared to school percentiles from 2012-2017 • Different comparison “universe” • Inclusion of additional indicators • Fewer years of data used in calculation • Use caution when comparing 2018 & 2019 percentiles • 2018 percentiles include only one year of data, while 2019 percentiles include two years of data • Percentiles should not be compared across gradespans • Compare a non-high school to another non-high school • Compare a non-high school to a middle/high or K-12 school

22. Criterion-referenced component 5

23. Criterion-referenced component – rationale • Federal law (ESSA) • Requires states to establish ambitious long-term goals & measures of interim progress • Perception • Accountability determination should not depend solely on the relative success of other schools • Resource allocation • Accountability system built solely on a normative measure (percentile) may not sufficiently differentiate schools

24. Criterion-referenced component • Focus on closing the achievement gap by raising the “achievement floor” • Gap-closing can occur as a result of a decline in performance by the high-performing group • In addition to meeting targets for the school as a whole, the performance of the lowest performing students in each school will be measured • Every school has a group of lowest performers • Identified from a group of students who have been enrolled in the school for more than one year

25. Lowest performing students – cohort model • For most schools serving grades 3-8, eligible students were: • Officially enrolled in current school for two consecutive years; • October 1, 2017 through October 1, 2018 (SIMS) • Tested in current school in 2018 & 2019 ortested in current school in 2019 & in current district in 2018; & • Not a first- or second-year English learner in 2019 • Not assessed using MCAS-Alt in both ELA & math • The lowest performing group represents the lowest performing 25 percent of eligible students • In schools where a legitimate cohort cannot be identified (fewer than 20 students), accountability results will be based on the performance of the “all students” group only

26. Identifying students in the lowest performing group – cohort model • Students in the 2019 lowest performing group were identified using a combined 2018 ELA & math average scaled score • DESE will share each school & district’s list via a secure dropbox each fall, after October SIMS data are submitted & certified by all districts

27. Lowest performing students – year-to-year approach • In high schools, the cohort model cannot be used • Improvement will be measured using a year-to-year approach based on the performance of the lowest performing 25 percent of eligible students: • Officially enrolled in current school for two consecutive years; • October 1, 2017 through October 1, 2018 (SIMS) • Tested in grade 10 in current school in 2019, & attended grade 9 in the same school or district in 2018; & • Not a first- or second-year English learner in 2019 • Not assessed using MCAS-Alt in both ELA & math • In schools where a legitimate cohort cannot be identified (fewer than 20 students), accountability results will be based on the performance of the “all students” group only

28. Identifying students in the lowest performing group – year-to-year model • Students in the 2019 lowest performing group are identified using a combined 2019 ELA & math average scaled score • DESE will share each school & district’s list of eligible students via a secure dropbox each fall, after October SIMS data are submitted & certified by all districts • Lowest performers are identified during accountability calculations in the summer (after testing)

29. Criterion-referenced component • Targets set for each accountability indicator, for the school as a whole & for the lowest performing students in each school *Minimum group size for each indicator = 20 students

30. Setting targets • For 2019 reporting, targets have been set for one year • Long-term targets will be set in the future • Targets for achievement indicators are based on the assessment performance of schools that have demonstrated improvement in the past • For example, the 25th percentile improvement of “improvers” on MCAS • Targets for non-assessment indicators are based on analysis of past trends & reasonable expectations for improvement

31. Criterion-referenced component Points assigned based on progress toward target for each indicator for the all students group, the lowest performing students group, & each subgroup with sufficient data:

32. Criterion-referenced component calculation – non-high school

33. Criterion-referenced component calculation – high school

34. Cumulative criterion-referenced calculation • Step 1: Calculate the annual criterion-referenced target percentage for each year separately (2018 & 2019) • Step 2: Calculate thecumulativecriterion-referenced target percentage • Districts, schools, or groups with two annual criterion-referenced percentages (2018 & 2019) will use a 40/60 weighting in the calculation of the overall criterion-referenced target percentages • In districts, schools, or groups with no 2018 annual percentage, the cumulative criterion-referenced target percentage will be the 2019 percentage (100%)

35. Assessment participation 6

36. Assessment participation • Districts & schools are required to maintain a 95% participation rate for the district/school as a whole & for each subgroup • Calculation includes: • ELA, math & science MCAS • ACCESS for ELLs (included in ELA rate) • Calculated two ways, with higher rate used in determination: • Actual 1-year rate for current year, & • Combined 2-year rate (if actual 1-year rate is <95%)

37. Assessment participation Calculate participation rates for all subjects combined, separately for each subgroup. Rate for “all students” group remains disaggregated by subject

38. Categorization of schools 7

39. Categorization of schools

40. Categorization of schools • Accountability determinations for schools will be based on 5 factors: • Accountability percentile (1-99) • Criterion-referenced target percentage (0-100%) • Subgroup performance (subgroup percentile 1-99) • Graduation rate (if less than 66.7%) • Assessment participation (if less than 95%) • Underperforming & chronically underperforming decisions made at the discretion of the Commissioner (designation & exit) • Schools ending in grade 3 will be classified based on criterion-referenced component only • No student growth, therefore no accountability percentile • Schools with no tested grades will be classified as “insufficient data”

41. Categorization of districts 8

42. Categorization of districts • Districts are classified based on the performance of the district as a whole (i.e., treated like one big school) • District accountability percentiles are not calculated • Classified based on criterion-referenced component • Adjustments made for low graduation rates & low assessment participation • Board may designate a district as underperforming or chronically underperforming

43. Categorization of districts

44. Reporting 9

45. Accountability reports • Accountability reports published for each district & school (fall 2019) • Reports include: • Overall classification • Including reason(s) for classification (e.g., low graduation rate, low-performing subgroup, participation) • Federal designation (if applicable, schools only) • Criterion-referenced target percentages (annual & cumulative) • Accountability percentile (schools only) • Data related to performance on each accountability indicator for each subgroup meeting the minimum group size (20 students) • All students • Lowest performing students • High needs students • English learners (ELs) & former ELs • Students with disabilities • Economically disadvantaged students • Major racial/ethnic subgroups

46. 2019 accountability reporting schedule • Preliminary & official embargoed accountability reports are in a separate Accountability Data application in the Security Portal (not in dropboxes). Appropriate security roles are assigned by district Directory Administrators: • Accountability (District Level) – to access district-level reports • Accountability (School Level) – to access individual school reports • Preliminary & official accountability data are confidential & may not be discussed publicly until DESE releases official results in late September • See DESE’s 2019 Assessment and Accountability Reporting Schedule for updated reporting dates

47. Resources 10

48. Accountability resources • http://www.doe.mass.edu/accountability/lists-tools.html • Public reporting schedule • Accountability system summaries • Detailed/technical documentation • Lists/data files • …and more!

49. Thank you! esea@doe.mass.edu · (781) 338-3550