Impact Evaluation Methods Regression Discontinuity Design and Difference in Differences

1. Impact Evaluation MethodsRegression Discontinuity Design and Difference in Differences Slides by Paul J. Gertler & Sebastian Martinez

2. Measuring Impact • Experimental design/randomization • Quasi-experiments • Regression Discontinuity • Double differences (diff in diff) • Other options

3. Case 4: Regression Discontinuity • Assignment to treatment is based on a clearly defined index or parameter with a known cutoff for eligibility • RD is possible when units can be ordered along a quantifiable dimension which is systematically related to the assignment of treatment • The effect is measured at the discontinuity – estimated impact around the cutoff may not generalize to entire population

4. Indexes are common in targeting of social programs • Anti-poverty programs  targeted to households below a given poverty index • Pension programs  targeted to population above a certain age • Scholarships  targeted to students with high scores on standardized test • CDD Programs  awarded to NGOs that achieve highest scores

5. Example: Effect of Cash Transfer on Consumption • Target transfer to poorest households • Construct poverty index from 1 to 100 with pre-intervention characteristics • Households with a score <=50 are poor • Households with a score >50 are non-poor • Cash transfer to poor households • Measure outcomes (i.e. consumption) before and after transfer

7. Non-Poor Poor

9. Treatment Effect

10. Case 4: Regression Discontinuity • Oportunidades assigned benefits based on a poverty index • Where • Treatment = 1 if score <=750 • Treatment = 0 if score >750

11. Case 4: Regression Discontinuity Baseline – No treatment 2

12. Case 4: Regression Discontinuity Treatment Period

13. Potential Disadvantages of RD • Local average treatment effects – not always generalizable • Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size • Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: • Nonlinear Relationships • Interactions

14. Advantages of RD for Evaluation • RD yields an unbiased estimate of treatment effect at the discontinuity • Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy • No need to “exclude” a group of eligible households/individuals from treatment

15. Measuring Impact • Experimental design/randomization • Quasi-experiments • Regression Discontinuity • Double differences (Diff in diff) • Other options

16. Case 5: Diff in diff • Compare change in outcomes between treatments and non-treatment • Impact is the difference in the change in outcomes • Impact = (Yt1-Yt0) - (Yc1-Yc0)

17. Outcome Average Treatment Effect Treatment Group Control Group Time Treatment

18. Outcome Average Treatment Effect Estimated Average Treatment Effect Treatment Group Control Group Time Treatment

19. Diff in Diff • Fundamental assumption that trends (slopes) are the same in treatments and controls • Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention)

20. Case 5: Diff in Diff

21. Impact Evaluation Example – Summary of Results