1 / 25

Regression Discontinuity Design Case Study : National Evaluation of Early Reading First

Regression Discontinuity Design Case Study : National Evaluation of Early Reading First Peter Z. Schochet. Decision Information Resources, Inc. Overview. ERF program Evaluation research questions Regression discontinuity design Conclusions. ERF Program.

tanaka
Download Presentation

Regression Discontinuity Design Case Study : National Evaluation of Early Reading First

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Regression Discontinuity Design Case Study: National Evaluation of Early Reading First Peter Z. Schochet Decision Information Resources, Inc.

  2. Overview • ERF program • Evaluation research questions • Regression discontinuity design • Conclusions

  3. ERF Program • Part of the No Child Left Behind Act • 3-year ERF grants provided to a collaboration of preschools • Funding focus is on low-income children • Goal: Enhance the language, cognitive, and early reading skills of preschool children

  4. ERF Funds Are Intended to: • Provide professional development for teachers • Create high-quality and print-rich environments • Promote the use of scientifically proven literacy methods and instructional materials • Identify preschool children at risk for reading failure

  5. Study Research Questions • What are the impacts of ERF on: • Children’s language and literacy? • Quality of language and literacy instruction, practice, and materials?

  6. KEY DESIGN FEATURES

  7. Study Focus Is on FY 2003 ERF Grant Applicants • 700 sites submitted pre-applications • 126 invited to submit full applications

  8. Random Assignment WasNot Possible • ED required that funds be awarded based on rankingsof applications • Applications were “scored” • 30 sites were funded with scores  74 • 96 unfunded sites with scores < 74 • Scoring criteria were set a priori • Favorable conditions for a RD design

  9. The Ideal ERF RD Design: Compare “73s” to “75s”: Almost Experimental Cutoff Mean for 75s Mean for 73s

  10. But There Are Not Enough 73s and 75s: Need to Include Other Sites ScoreNumber of Sites 42 to 53 22 54 to 63 21 64 to 73 21 Cutoff Value=74 74 to 83 18 84 to 95 12 UNFUNDED FUNDED

  11. Sampling • All funded sites agreed to participate • Sorted unfunded sites by their scores • Sites with largest scores contacted first • 64 sites contacted • 37 agreed to participate • Obtained lists of classrooms in sites • Sampled 3 classrooms per site • Selected 9 four-year olds per classroom • 94 percent parental consent rates

  12. The RD Method Visually Estimated Regression Lines Funded Unfunded Impact Cutoff

  13. Key Identifying Assumption • There must be a continuous relationship between the outcome measure and the application score

  14. Differences-in-Means Estimates Could Be Biased = Simple Means Funded Unfunded Spurious Impact

  15. RD Designs Require Larger Samples Than Experimental Designs • Controlling for the application score reduces power • Design effects are 3 to 4

  16. The Correct Functional Form Specification Is Crucial for Obtaining Unbiased Estimates Assume Linearity When the True Relationship is Nonlinear Spurious Impact

  17. There Also Has to Be a Clear Functional Form Relationship % of Classrooms in Site That Engage in an Activity

  18. Basic Regression (HLM) Model • Y = β0+ β1*T + β2*f(SCORE-74) + u • Y = Outcome measure (child or teacher level) • T = 1 if funded site, 0 if unfunded • f(SCORE) = Function of application score • β1 = Impact estimate • u = Error term accounting for site and classroom-level clustering

  19. Selecting f(SCORE) • Graph Y on SCORE • Add SCORE2, SCORE3, and T*SCORE interaction terms and test for significance • Use nonparametric methods • Specification tests • Impacts = 0 using baseline data • Impacts = 0 at “artificial” cutoff values • Adding covariates should not change impacts

  20. Interpretation of Impact Estimates • In the “73-75” model, results pertain only to sites around the cutoff value • Results “generalize” more broadly using the parametric approach • Does this stretch the results too far? • But not using the nonparametric approach

  21. Conclusions • RD designs can produce rigorous impact estimates under the right conditions: • Need exogenous “scores” • Scores and outcomes must have a smooth relationship that can be credibly modeled • But there are limitations to the RD approach: • Need larger samples than an experimental design • Generalizability • Nonresponse a problem in unfunded sites

  22. EXTRA SLIDES

  23. Data: Fall 2004 and Spring 2005 (Spring Response Rates) • Child assessments (97%) • Teacher behavioral ratings (96%) • Teacher/classroom observations (79%) • Parent surveys (69%) • Teacher surveys (91%) • Center director interviews (88%)

  24. Child Assessment Instruments

  25. Observations and Surveys

More Related