Using School Choice Lotteries to Test Measures of School Effectiveness

1. Using School Choice Lotteries to Test Measures of School Effectiveness David Deming Harvard University and NBER

2. Measuring School Effectiveness • School rankings, ratings, league tables • Gain score or “value-added” modeling approach (VAMs) • School VAMs now in ~30 U.S. states (Blank 2010) • Teacher VAMs used in evaluation, retention • Accuracy of VAMs is important for incentive design and student welfare (Baker 2002, Rothstein 2010)

3. VAM Research • Large literature on measurement / technical issues • First order issue mostly untested - is assignment of teachers to classes within schools as good as random? • Kane and Staiger (2008), Kane et al (2013) randomly assign classes to teachers, test validity of VAMs • Chetty, Friedman and Rockoff (2013) use quasi-experimental design with teacher mobility • School VAMs require conditionally exogenous sorting of students across schools • Would you consent to this experiment?

4. Using School Choice Lotteries • Oversubscribed public schools in Charlotte-Mecklenburg • Random assignment, within a self-selected group of applicants • Estimate VAMs using data from prior cohorts • Vary model specification, sample, outcome • Out-of-sample predictions of “school effects” • Use VAM estimates to predict the treatment effect of winning the lottery

5. Data and VAMs • Grades 4-8 • Covariates - test scores from 1996-97 to 2001-2002, demographics • Models with nothing in X (levels), lagged scores only, lagged scores + demographics • Outcomes - 2003 and later test scores ; • Obtain by computing average residual (random effects) • Results are very similar with fixed effects • “School-by-grade” effects

6. Lottery Data and Sample • 2,599 students in 118 separate lotteries for “marginal” priority groups • Top 3 choices but nearly all randomization was over 1st choices

7. Empirical Strategy 2SLS with VAM in Fall 2002 school as the endogenous variable: 1 = 1st choice school N = neighborhood school = indicator for winning lottery j = lottery fixed effects (unit of randomization) If = 0.1, implied causal effect of attending school j relative to average school is 0.1. Thus if is unbiased, = 1. < 1 means VAM is upward biased… isn’t obvious - what is lottery applicant’s outside option? We try a few different things to check this...

8. 4 Possible Explanations for Bias • Sorting on unobserved determinants of student achievement (Rothstein 2010) • Estimation error • “True” school effects may vary over time independent of estimation error • Lottery sample is self-selected, so treatment effect is different for them

9. No correlation between average test scores (in levels) and lottery impacts

10. Huge improvement from adding lagged scores (gains model) • Need 2+ years of data to fail to reject unbiasedness (triple negative!)

11. Adding demographics to increases coefficients modestly in all specifications • School effects “too small” when all prior years of data are included

12. “Shrinkage” • Attenuate teacher/school effects toward zero based on transitory variance in the estimate • Estimate a constant from noisy data • Empirical Bayes (e.g. Kane et al 2013) weighs all past years of data equally • Weigh based on autocovariance – teacher effects “drift” over time (Chetty, Friedman and Rockoff 2013)

13. Shrinkage improves prediction when only one year of prior data is used – with longer panel unshrunken more accurate • “Drift” adjustment is best here • Shrinkage overcompensates if there is true variation in school effects – CFR call this “teacher bias”

14. Concluding thoughts • Despite sorting concerns, school VAMs are surprisingly accurate • Best fit was gains only on full panel, but that’s not a theorem • Large standard errors admit non-trivial bias – need to replicate in other settings • Good news for policies that use VAMs, but: • Biases might be offsetting • Other outcomes are important too! • “School effects” may include contextual factors that are beyond school’s control (peers, neighborhoods)

15. Thanks!