Making the Most out of Discontinuities

1. Presented by Malte Lierl (Yale University) Making the Most out of Discontinuities

2. Introduction • How do we measure program impact when random assignment is not possible ? • e.g. universal take-up • non-excludable intervention • treatment already assigned • Solutions • Make assumptions about what constitutes a plausible control group (matching on observables, diff-in-diff) • Exploit quasi-random aspects of program implementation • Quasi-experiments • Example: Regression Discontinuity Design (RDD)

3. Discontinuity = Arbitrarily placed cutoff for program eligibility Regression Discontinuity Design (RDD) income income vulnerability index vulnerability index PROGRAM IMPACT CUTOFF • Around the cutoff, beneficiary (‘treated’) and non-beneficiary (‘untreated’) populations are very similar. • For the population around the cutoff, RDD can be as credible as a randomized experiment.

4. RDD: Some examples Example 1: Evaluate reintegration assistance for former child soldiers aged 16 and below. An ex-combatant aged 16 years and one day would not benefit from the program. RDD would compare individuals just above and just below 16 years of age.

5. RDD: Some examples Example 2: If you are elected into parliament, will this make you wealthier? Can’t randomize who gets into parliament. In majoritarian systems such as in the UK, you get into parliament if you have the majority of votes in a district. Some districts have very close election results. Between two candidates with 49.5% and 50.5% of votes it is as good as random who gets into parliament. RDD: compares winners and losers in very close runoffs.

6. Another RDD example Example 3: Minimum legal drinking age in the United States is 21 It is illegal to sell alcohol to people younger than 21 People aged 21 and people aged 20, 11 months, 29 days are treated very differently under the drinking age policy But they are not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc.)

7. What is the effect of alcohol on mortality rates? In effect, the minimum drinking age assigns people into ‘treatment’ and ‘comparison groups’ Treatment group: People between ages 20 years and 11 months and 20 years 11 months and 29 days cannot drink alcohol. Comparison group: People just above 21 can drink. Both groups should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates). If we use the drinking age cutoff as RDD, we can estimate the causal impact of alcohol consumption on mortality rates among young adults.

8. What is the effect of alcohol on mortality rates? Proportion of days drinking, by age RDD Source: Carpenter & Dubkin, 2009

9. What is the effect of alcohol on mortality rates? Death rates, by age Increased alcohol consumption causes higher mortality rates around the age of 21 All deaths All deaths associated with injuries, alcohol or drug use RDD All other deaths Source: Carpenter & Dubkin, 2009

10. Internal Validity If the cutoff is arbitrary: Individuals directly above and below the cutoff should be very similar in expectation Systematic differences in outcomes are caused by the policy Major assumptions: Individuals have no precise control over assignment variable Nothing else is happening. In absence of the policy, we would not observe a discontinuity around the cutoff. Might not be the case if: Drinking age is 18, and driving also becomes legal at age 18 Another program provides reintegration assistance for ex-combatants over 16 years.

11. RDD Requirements Transparency and precise knowledge of the selection process ‘Treatment’ is discontinuous with respect to an assignment variable Individuals cannot precisely manipulate the assignment variable All other factors are continuous with respect to the assignment variable (“nothing else is happening”) Enough data points around the cutoff

12. Sharp and Fuzzy RDDs • Sharp discontinuity • Discontinuity precisely determines treatment status • All people 21 and older drink alcohol and no one else does • All ex-combatants younger than 16 receive assistance, nobody else does • Fuzzy discontinuity • Percentage of participants changes discontinuously at cut-off, but not from 0% to 100% (or from 100% to 0%) • Some people younger than 21 end up consuming alcohol and/or some older than 21 don’t consume at all • Some youth ex-combatants under 16 don’t participate, and their slots are given to others who are just over 16.

13. Sharp and Fuzzy RDDs FUZZY DISCONTINUITY SHARP DISCONTINUITY Probability of being treated Probability of being treated 1 1 0 0 assignment variable assignment variable

14. External validity Are RDD estimates of program impact generalizable? Counterfactual/control group in RDD: Individuals marginally excluded from benefits Examples: Ex-combatants over 16, candidates with 49.5% of votes Causal interpretation is limited to individuals/households/villages near the cutoff Extrapolation beyond this group needs additional (often unwarranted assumptions) Or multiple cutoffs!

15. RDD Implementation Data collection: Make sure to have enough observations around the cutoff Analysis: Observations away from the cutoff should have less weight outcome weight assignment variable • Why? • Only near the cutoff can we assume that people find themselves to the left and to the right of the cut-off by chance.

16. RDD Implementation Carefully justify study design Baseline data will be useful to verify assumptions BEFORE PROGRAM AFTER PROGRAM outcome outcome assignment variable assignment variable

17. RDD Implementation Carefully justify study design Graphical analysis is an important tool outcome assignment variable

18. Summary Advantages of RDDs: RDD can be applied even when randomization is not feasible e.g. to programs with means tests for eligibility For the population around the cutoff, RDD is as credible as a randomized experiment Requires fewer assumptions than other non-experimental methods RDD can be used like a ‘natural experiment’ to evaluate a program ex-post

19. Summary Drawbacks of RDDs: Limited external validity: The estimates of program effects are informative only for the population around the cutoff. RDD requires a lot of data around the cutoff Knowledge about the cutoff may induce behavioral change that can bias your evaluation e.g. ex-combatants misreport their age e.g. candidates become frustrated because they were ‘so close’ to getting elected

20. Thank you! شكرا Further reading: Lee, David and Thomas Lemieux (2009): Regression Discontinuity Designs in Economics, NBER Working Paper No. 14723. http://www.nber.org/papers/w14723