Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under NCLB

1. October, 2012 Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under NCLB Randall Reback Barnard College, Columbia University Jonah E. Rockoff Columbia Business School Heather L. Schwartz RAND Excellent research assistant provided by Tamara Lalovic Cox & Elizabeth Davidson Funding provided by the Spencer Foundation and the U.S. Institute of Education Sciences

2. Policy Background: No Child Left Behind • First took effect during spring of 2003 • Requires states to adopt school accountability systems that determine whether public schools satisfy Adequate Yearly Progress (AYP) • Schools’ AYP status based on • % of students taking and demonstrating proficiency on statewide exams in math and reading • both overall and student subgroups pass rates • state-specific exams, standards, and rules • Consequences of failing AYP • Escalating sanctions, including • Inter-district public school choice • Funds redirected to after-school tutoring for students from low income families • States are also required to publish annual school report cards, so schools’ AYP status can also affect school prestige and local property values.

3. Overview of Our Study • Assemble NCLB related data and outcomes for all schools nationwide for first 2 years of NCLB • Find schools that were at substantial risk of failing AYP and therefore faced pressure under NCLB • Considerable variation across states • Use several external data sets to investigate the impact of NCLB pressure on teachers and students

4. Preview of Main Results • Teacher-level results: Accountability pressure • increases teachers’ concerns about their job security and decreases their expected career length • decreases frequency of instruction in low-stakes subjects (e.g., science) and decreases time on whole-class instruction • Student-level results In schools facing the strongest short term accountability incentives, students… • score higher on low-stakes readings exams • perform at least as well on low-stakes math and science exams • do not experience any negative effects in terms of their enjoyment of math & reading or their anxiety about testing

5. NCLB Policy Variation Interaction of four features significantly influences the likelihood that a school fails AYP: • state rules for the numerical significance of student subgroups • within-school heterogeneity, which influences how many student subgroups are numerically significant • the generosity of the state’s confidence intervals • the generosity of the state’s safe harbor provisions Several other idiosyncratic policies also matter

6. Methodology (Part 1) • Predicting which schools were on the margin of making AYP • Use 2002 data on test scores and demographics (pre-policy) to predict probability of making AYP in 2003 and 2004 using separate probit regressions for each state • Define schools as on the AYP margin if… …at least one group moderate probability of passing …no group has a very low probability of passing • Define schools as below the AYP margin if any group has a very low probability of passing

7. Difference-in-differences, An Example • Take two pairs of schools, from NJ and PA • Even though pairs are observably similar, differences in state rules create arguably exogenous variation in NCLB pressure

8. Difference-in-differences Regressions control for • state fixed effects • various school-,child-,household- level controls • including student-level and school-level pre-NCLB test performance • simulated % of states where school would be on AYP margin • simulated % of states would be below margin

9. Evidence from RAND-NCLB Survey of Math Teachers

10. Evidence from RAND-NCLB Survey of Principals

11. Data Used in Second Stage Analysis • Early Childhood Longitudinal Survey (ECLS): students in the kindergarten cohort of 1998-99, followed up at 1st, 3rd, 5th (and 8th) grade • Nationally representative, but not from all 50 states • Rich data set with low stakes exams. Used adaptive testing to capture wide range of abilities. • SASS 2003-04: nationally representative sample of schools and teachers within schools • Restricted-use data allow use to link ECLS/SASS observations with variables on NCLB pressure

12. Effects of NCLB on Teachers

13. Effects of School being on AYP Margin on Student Learning

14. ECLS: Other Findings • Effects are not very different for • Students in the subgroups under pressure • “Bubble students” • Students from low-income families • Stronger positive effects for math score gains in states without strong accountability prior to NCLB (related to Dee and Jacob, 2011)

15. Conclusions • States vary widely in rates of making AYP • Cross-state variation in student academic aptitude or in exam difficulty explains relatively little of this variation • Short term NCLB pressure… • Threatens perceived job security of teachers • Influences teachers’ time use • Has positive net effects on average student test score growth on low-stakes reading exams • Does not have any short-term negative effects on outcomes for the average student!

16. No Data Left Behind! Our NCLB data is publicly available from our “No Data Left Behind” website: http://www7.gsb.columbia.edu/nclb/

18. First stage: Pr(Made AYP in ’03 & ’04) • Large variation across groups in contributions to risk of AYP failures • 1 in 5 marginal; 1 in 10 low probability • Large differences in actual AYP outcomes

19. Which Children Have Been Left Behind?