EVAL 6970: Experimental and Quasi-Experimental Designs

1. EVAL 6970:Experimental and Quasi-Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012

2. Agenda Regression discontinuity designs

3. Questions to Consider • Throughout today’s discussion, consider the following questions • What characteristics of regression discontinuity designs make them more amenable to causal interpretation versus those discussed previously (i.e., quasi-experimental designs with and without pretests and control groups and interrupted time-series)? • What are the associated limitations or drawbacks of regression discontinuity designs, if any? • How could you apply regression discontinuity designs to your own work?

4. Note • The principles of regression discontinuity can be confusing, so… • …ASK QUESTIONS!!!

5. Basic Structure of Regression Discontinuity Designs

6. Basic Structure

11. Theory of Regression Discontinuity

12. Some Assumptions Most randomized experiments compare posttest means Regression discontinuity designs compare regression lines for treatment and control groups Rather than making the assumption that pretest means are equivalent for both groups, regression discontinuity looks for a change between the functional form of the regression line (e.g. slope or intercept) for the treatment group and control

13. Some Assumptions In (most) other designs for quasi-experiments, the selection process is never fully known (selection is determined by a large system of variables beyond the researcher’s control) In sharp contrast, there are no unknowns in the assignment process for regression discontinuity In fact, the selection process is completely known and perfectly measured (even though there will be some error associated with the assignment variable) Assignment variables only measure how participants got into conditions, and when assignment is based only on that score, error is effectively zero

14. Some Assumptions • Additionally, most experiments and quasi-experiments try to equate treatment and control • Regression discontinuity designs explicitly acknowledge—and, in fact, base assignment on—pre-existing differences between treatment and control groups • Units are assigned to conditions based on a cutoff score (i.e., cut score) on an assignment variable • The assignment variable must occur prior to treatment • Units on one side of the cut score are assigned to one condition, and units on the other side to another condition

15. Implementing Regression Discontinuity Designs

16. Implementation • Like randomized experiments, regression discontinuity designs yield unbiased estimates of treatment effects (as long as assumptions are met) • Assignment to treatment must be based only on the cutoff score • The assignment variable cannot be caused by the treatment, and must be continuous • Dichotomous variables should not be used as an assignment variable, because this makes it impossible to estimate a regression function • The assignment variable is often a pretest score, but it can be almost anything

17. Implementation • The cut score should be near the mean of the assignment variable, because extreme values can cause problems • Modeling the regression function can become more difficult and/or error-prone in smaller samples • Statistical power depends on sample size, and also “prefers” samples with equal or nearly equal sizes • Researchers can use a composite variable for assignment, to include the effect of multiple influences • Avoiding selection bias requires that • Assignment to conditions is strictly controlled • All units could have received treatment

18. Forms of Effects

19. Types of Effects • First, note that “discontinuity” means • A treatment effect (if present) causes an upward or downward displacement in the regression function • A discontinuity can be a change in either the intercept or the slope • The discontinuity should occur at exactly the cut score

20. Threats to Validity

21. Validity Threats • Few internal validity threats are plausible with a well planned and correctly executed regression discontinuity design • A plausible threat would have to cause a discontinuity in the regression line that corresponds precisely with the cut score • Except in rare cases, selection, history, and maturation threats are drastically reduced • However, attrition can be a major concern, especially if attrition rates are correlated with the assignment variable • Therefore, validity concerns mainly focus on statistical conclusion validity • In particular, the regression lines must be good models of functional form (e.g., nonlinear functions, interaction terms)