1 / 30

Regression Discontinuity Design

Regression Discontinuity Design. Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides. RD Designs. A pretest- posttest , program-comparison group strategy Review: Advantages of Pre-tests ? Detect differences between groups

magnar
Download Presentation

Regression Discontinuity Design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Regression Discontinuity Design Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides

  2. RD Designs • A pretest-posttest, program-comparison group strategy • Review: • Advantages of Pre-tests? • Detect differences between groups • Detect potential vulnerability to internal validity threats • Helps with statistical analysis • Advantages of Comparison groups? • Helps control sources of error • Helps support the counterfactual inference

  3. Underuse of RD? Why? • It’s new. • Key criteria must be met for use. • Perhaps it’s just misunderstood.

  4. Overview of RD OA C X O2 OA C O2 • “pre” is ANY continuous variable that correlates with the outcome of interest • Assignment based on cutoff score • Regression line should have vertical displacement at the cutoff score if there is an effect

  5. No Treatment Effect

  6. Positive Effect

  7. Examples • Campell & Stanley’s Ivy League Education Example • Trochim’s Hospital Administration Example Hospital Quality of Care

  8. More about assignment • Assignment variables: • Must be continuous (or ordinal) • Can be a pretest on a dependent variable • Can be by order of entry into study • Cannot be caused by treatment • May or may not be related to the outcome If implementing an RD design in your area of research, what variables would you choose for assignment?

  9. Choosing the Cutoff Score • Now referring to the assignment variable(s) you identified, how would you arrive at a cutoff score? • Substantive grounds: professional judgment • Need or Merit • Clinical diagnosis • Practical grounds: • Available data sets • Available resources

  10. Choosing the Cutoff Score • Mean of the distribution of assignment scores • Politically defined thresh-holds • Composite scores of assignment variables Important: Assignment must be controlled! (It is as important as proper random assignment.)

  11. Additional Considerations • Functional form relating the assignment and outcome variables • A defined population in which it is possible for all units in the study to receive Tx regardless of the choice of a certain cutoff point • Intent-to-Treat? : Tx diffusion and cross-over participants

  12. Variations • Compare 2 treatment groups • Compare 3 conditions • Different dose treatment groups • Multiple cutoff points • …and many more creative ways to think of

  13. Theory of RD – How does this work? • RDs as Treatment Effects in REs • RE pretest means of Tx and Control groups nearly identical at would be the cutoff score in an RD design through random assignment • Cutoff-based assignments creates groups with different pretest means and non-overlapping pretest distributions • RD compares regression lines, not means • Both RDs and REs control for selection bias • Unknown variables do not determine assignment • Pretests have no error IF used as the selection variable • Regression lines are not affected by posttest errors

  14. Adherence to the Cutoff • Overrides of the cutoff • Crossovers • Attrition • “Fuzzy” regression discontinuity

  15. Threats to Validity • Statistical Conclusion Validity • Nonlinearity • Interactions • Internal Validity – must occur at the cutoff point • History • Maturation • Mortality • Selection-instrumentation

  16. Interaction

  17. Group Exercise: RD Design

  18. Analytical Assumptions • No exceptions to the cutoff • Adhere to true function of the pre-post relationship • Uniform delivery of pretest and program

  19. Combining RD with Randomized Experiments 7 combo examples 3 advantages: • Increased power • Allows estimation of both groups at the overlap interval • Adds clarity to the cutoff point

  20. RD – Quasi-experiment? • shortfalls are not yet clear • Requires more “demanding statistical analysis” • Less statistical power • see table 7.2 in SCC (pg. 243)

  21. Analysis Problems • The Curvilinear Problem

  22. Steps to Analysis • Transform the pretest • Examine the relationship visually • Specify high order terms and interactions • Estimate the initial model • Refine

  23. Comparing RD Designs with Experimental Designs • IN theory, both designs should produce similar results when all exemplary conditions of each method type exist • Question remains do they produce similar results and standard errors in practice (real world settings)? • Under exemplary conditions, experiments are 2.75 times more efficient than RDDs • If otherwise, the degree of this efficiency will vary • Central Question: How to compare these two design options in field settings? Cook, Shadish& Wong 2008

  24. Statistical Power for GRT and RDD • the RDD has approximately 36 per cent the efficiency of the GRT. • This implies that the RDD will require approximately 2.75 times more groups than a GRT with the same power. • The same result was found by Schochet [16] for hierarchical models in education • and by Cappelleri and Trochim [31] for trials targeting individuals rather than groups. Pennell et al., 2010

  25. Within Study Comparisons: • Proposed methodology from LaLonde • Causal estimates derived from an experiment compared with estimates from a non-experiment • Same Tx Group • Different Control Group • Modifications needed to use for RDDs

  26. 7 Criteria to Improve Interpretation of Within-Study Comparisons • Must demonstrate variation in types of methods being contrasted • Both assignment mechanisms cannot be correlated with other factors related to outcome variables • The RE must “deserve” its status of the causal “Gold Standard” • The non-experiment design must also be good AND

  27. 7 Criteria to Improve Interpretation of Within-Study Comparisons • Both study types should estimate the same causal quantity • Explicit criteria must be raised on how the two design estimates relate to each other • Blind that data analyst! AND

  28. Further Discussion? Nagging Questions? …or Inspirations?

More Related