The District-Level Achievement Impacts of Benchmark Assessments: Year 1 Outcomes of CDDRE

1. The District-Level Achievement Impacts of Benchmark Assessments: Year 1 Outcomes of CDDRE Geoffrey D. Borman, Deven Carlson, and Michelle Robinson, University of Wisconsin—Madison Presentation Prepared for IES Research Conference National Harbor, MD June 30, 2010

2. Introduction • Student assessments, accountability programs, and the use of associated data systems have recently emerged as central strategies for improving the nation’s public schools • Impetus for such strategies can be traced to NCLB • Relatively little rigorous empirical work evaluating the impacts of adopting more proactive uses of data and data systems on student and school achievement • CDDRE is one of the most prominent research and development efforts in this area

3. Research Question • This presentation reports the results of an experimental evaluation of one component of the CDDRE intervention • Specifically, we report results from a multi-state district-level cluster randomized trial of the impact of quarterly benchmark assessments on reading and mathematics achievement • Research Question: • Does the implementation and administration of benchmark assessments, coupled with the provision of consulting services to assist in the interpretation of the resulting data, bring about district-wide changes in student achievement relative to a control condition in which districts operated as usual without benchmark assessments and associated services?

4. Existing Literature • Data-driven reform involves collecting, interpreting, and disseminating data in a manner that is intended to inform and guide district and school improvement efforts. • Bernhardt (2003) identifies four categories of data that practitioners can analyze to inform reform efforts • Student learning and assessment • Demographics • School Processes • Teacher Perceptions • Perie, Marion, & Gong (2009) discuss three possible uses of assessment results • Instructional • Evaluative • Predictive

5. Existing Literature • Empirical literature on data-driven reform can be classified into two main groups of studies • Teacher responses to provision of student data • Effects of data-driven decision-making on student outcomes • Teacher response to provision of student data • Teachers believe interim tests assist them in identifying areas in which their students would benefit from additional instruction • Teachers report that they alter instruction in response to assessment results • Results come from teacher self-reports • Objective assessments of teacher data use reach somewhat less optimistic conclusions

6. Existing Literature • Effects of data-driven decision-making on student outcomes • Three large-scale empirical studies • Quint, Sepanik, & Smith (2008) evaluated a data-driven instruction program, which included benchmark assessments, implemented in 21 public schools in Boston during 05-06 and 06-07 school years • No statistical or practical differences in achievement • Henderson et al. (2007) examined effect of benchmark assessments in 8 MA school districts • No statistically significant effects. Point estimates were positive, but study was somewhat underpowered • May and Robinson (2008) RCT evaluated Ohio’s Personalized Assessment Reporting System for Ohio Graduation Test • No effects for first-time test takers, but positive effects for re-takers • Several in-depth case studies attribute achievement improvements to data use

7. Existing Literature • On the whole, there is some empirical evidence that implementation of data-driven reforms, including benchmark assessments, can lead to improvement in achievement, but the evidence is far from conclusive • A group of RAND researchers note that more research on the topic is clearly needed: “...experimental studies are needed to more rigorously measure the effects of enhanced provision of data and supports to use it. Standardized interventions can be developed and tested in randomized trials. For example, studies might examine whether the provision of interim progress test data or value-added measures, combined with ongoing professional development for teachers on how to use the information, leads to better instruction and higher achievement than do classrooms without such data and training” (Marsh, Pane, & Hamilton, 2006, p. 12).

8. Method: Sample Recruitment • Sample recruitment took place over three school years— 2004-05 to 2006-07—across seven states: AL, AZ, IN, MS, OH, PA, and TN • State departments of education nominated districts with large numbers of low-performing schools to participate • Resulted in 59 districts agreeing to participate • District officials decided which schools to target—generally a subset of the lowest-performing schools in the district • Final sample included 549 schools across 59 districts • Sample has geographic, racial/ethnic, and socioeconomic diversity

9. Method: Randomization Procedures • District-level randomized block design • For each cohort, districts were blocked by state and then randomly assigned to proximate treatment condition (treatment group) or delayed treatment condition (control group) with selection probability of approximately 50 percent • Several advantages to the randomized block design • Intervention will be distributed in fair and equitable manner across states • Enhances the face validity of the study • Increases statistical precision and power • No statistically significant differences between treatment and control group on any pretreatment covariates

10. Results of Randomization

11. Results of Randomization

12. Method: Implementation of Treatment and Control Conditions • CDDRE intervention includes several components implemented across a three-year period • We test the impacts of the Year 1 components • Quarterly benchmark assessments tied to state standards and assessments in reading, writing, and mathematics—predict performance on state assessments, determine needs for specific interventions, and evaluate students’ progress toward state goals • Reviews of state test data, benchmark assessments, questionnaires from educators at all levels, and other indicators to identify areas of need for schools not meeting AYP goals or at risk for AYP failure • Training for state, district, and building leaders in interpreting and using data, managing resources to focus on areas of need, and leading a data-driven reform process. • “Business as usual” counterfactual during one-year treatment delay

13. Method: Outcome Measure • Outcome measure is school-level performance on state-administered achievement tests • Advantage: Every student tested at state-expense and grade-level or school-level data are generally publicly available • All data in this analysis were collected from state websites • Advantage: 4sight benchmark assessments developed from state assessment blueprints so state assessments are the ideal outcome measure • Disadvantage: Comparability across grades, subjects, and especially states can be difficult • Solution: Transform all assessment results into z-scores • Separate analyses for reading and math

14. Method: Covariates • Baseline school-level achievement • Increase the precision of impact estimates • Also in the form of z-scores • School-level demographics • Percent minority • Percent eligible for free- or reduced-price lunch • District-level demographics • Percent minority • Percent eligible for free- or reduced-price lunch • Percent receiving special education services

15. Method: Missing Data Procedures • Reading analysis • School-level achievement outcomes unavailable for 25 of 549 schools in our sample (approximately 5%) • Eight schools in treatment districts, 17 schools in control districts • Excluded from the analysis • Five schools with outcome data missing baseline achievement data • We follow guidance of Puma et al. (2009) and assign average achievement in district to these schools and include dummy code in analytic model • Three schools with outcome data missing free- or reduced-price lunch eligibility data • Assign average free/reduced lunch eligibility in district to these schools and include dummy code in analytic model • Analytic sample: 524 schools located in 59 districts

16. Method: Missing Data Procedures • Math analysis • Three districts did not administer benchmark assessments in math—Neither OH district and SE Delco in PA • Reduces number of schools in math analysis to 538 • School-level achievement outcomes unavailable for 29 of 538 schools (approximately 5%) • Seven schools in treatment districts, 22 schools in control districts • Excluded from the analysis • Three schools with outcome data missing baseline achievement or free/reduced lunch data • As in reading analysis, we assign average district value and include dummy in analytic model • Analytic sample: 509 schools located in 56 districts

17. Method: Analytic Model • Multilevel modeling approach • Appropriate for the analysis of data from cluster randomized trials such as this one • We estimate four models for each subject • Unconditional model to determine amount of variation in school-level achievement attributable to school-level factors versus district-level factors • Model with only treatment indicator and randomization blocks • Model with treatment indicator, randomization blocks, and baseline achievement • Fully specified model: treatment indicator, randomization blocks, baseline achievement, and baseline demographics

18. Method: Analytic Model • Fully specified level-one (school-level) model is written as: Yij = β0j + β1(Base Ach.)ij + β2(FRPL)ij + β3(Pct. Min.)ij + β4(Base Ach. Miss)ij + β5(FRPL Miss)ij + εij • Y represents average school-level achievement and i and j index schools and districts, respectively

19. Method- Analytic Model • Fully specified level-two, district-level, model written as: β0j = γ00 + γ01(Treat)j + γ02(Pct. Min.)j + γ03(FRPL)j + γ04(Pct. Sp. Ed.)j + δ(Rand. Block)j + τj • Treatment indicator at district level • Randomization blocks modeled as fixed effects

20. Results: Mathematics

21. Results- Reading

22. Results: Summary and Magnitude of Effects • Fully specified model indicated a treatment impact on mathematics achievement of d = 0.21 • Comparable to reducing school-level free/reduced lunch eligibility by 60 percentage points • In reading, the impact approached, but did not reach, conventional levels of significance (p-value = 0.10) • Equivalent to an effect size of d = 0.14 • Comparable to reducing school-level free/reduced lunch eligibility by 35 percentage points

23. Conclusions • One of the first large-scale, district-level experimental evaluations of benchmark assessments • Results suggest that implementation and administration of benchmark assessments, coupled with appropriate professional development, can raise student achievement, particularly in math • Geographic and socioeconomic diversity of our sample gives us confidence that the results of this CRT are generalizable