Regression Discontinuity Design Case Study : National Evaluation of Early Reading First

1. Regression Discontinuity Design Case Study: National Evaluation of Early Reading First Peter Z. Schochet Decision Information Resources, Inc.

2. Overview • ERF program • Evaluation research questions • Regression discontinuity design • Conclusions

3. ERF Program • Part of the No Child Left Behind Act • 3-year ERF grants provided to a collaboration of preschools • Funding focus is on low-income children • Goal: Enhance the language, cognitive, and early reading skills of preschool children

4. ERF Funds Are Intended to: • Provide professional development for teachers • Create high-quality and print-rich environments • Promote the use of scientifically proven literacy methods and instructional materials • Identify preschool children at risk for reading failure

5. Study Research Questions • What are the impacts of ERF on: • Children’s language and literacy? • Quality of language and literacy instruction, practice, and materials?

6. KEY DESIGN FEATURES

7. Study Focus Is on FY 2003 ERF Grant Applicants • 700 sites submitted pre-applications • 126 invited to submit full applications

8. Random Assignment WasNot Possible • ED required that funds be awarded based on rankingsof applications • Applications were “scored” • 30 sites were funded with scores  74 • 96 unfunded sites with scores < 74 • Scoring criteria were set a priori • Favorable conditions for a RD design

9. The Ideal ERF RD Design: Compare “73s” to “75s”: Almost Experimental Cutoff Mean for 75s Mean for 73s

10. But There Are Not Enough 73s and 75s: Need to Include Other Sites ScoreNumber of Sites 42 to 53 22 54 to 63 21 64 to 73 21 Cutoff Value=74 74 to 83 18 84 to 95 12 UNFUNDED FUNDED

11. Sampling • All funded sites agreed to participate • Sorted unfunded sites by their scores • Sites with largest scores contacted first • 64 sites contacted • 37 agreed to participate • Obtained lists of classrooms in sites • Sampled 3 classrooms per site • Selected 9 four-year olds per classroom • 94 percent parental consent rates

12. The RD Method Visually Estimated Regression Lines Funded Unfunded Impact Cutoff

13. Key Identifying Assumption • There must be a continuous relationship between the outcome measure and the application score

14. Differences-in-Means Estimates Could Be Biased = Simple Means Funded Unfunded Spurious Impact

15. RD Designs Require Larger Samples Than Experimental Designs • Controlling for the application score reduces power • Design effects are 3 to 4

16. The Correct Functional Form Specification Is Crucial for Obtaining Unbiased Estimates Assume Linearity When the True Relationship is Nonlinear Spurious Impact

17. There Also Has to Be a Clear Functional Form Relationship % of Classrooms in Site That Engage in an Activity

18. Basic Regression (HLM) Model • Y = β0+ β1*T + β2*f(SCORE-74) + u • Y = Outcome measure (child or teacher level) • T = 1 if funded site, 0 if unfunded • f(SCORE) = Function of application score • β1 = Impact estimate • u = Error term accounting for site and classroom-level clustering

19. Selecting f(SCORE) • Graph Y on SCORE • Add SCORE2, SCORE3, and T*SCORE interaction terms and test for significance • Use nonparametric methods • Specification tests • Impacts = 0 using baseline data • Impacts = 0 at “artificial” cutoff values • Adding covariates should not change impacts

20. Interpretation of Impact Estimates • In the “73-75” model, results pertain only to sites around the cutoff value • Results “generalize” more broadly using the parametric approach • Does this stretch the results too far? • But not using the nonparametric approach

21. Conclusions • RD designs can produce rigorous impact estimates under the right conditions: • Need exogenous “scores” • Scores and outcomes must have a smooth relationship that can be credibly modeled • But there are limitations to the RD approach: • Need larger samples than an experimental design • Generalizability • Nonresponse a problem in unfunded sites

22. EXTRA SLIDES

23. Data: Fall 2004 and Spring 2005 (Spring Response Rates) • Child assessments (97%) • Teacher behavioral ratings (96%) • Teacher/classroom observations (79%) • Parent surveys (69%) • Teacher surveys (91%) • Center director interviews (88%)

24. Child Assessment Instruments

25. Observations and Surveys