Difference in Difference and Regression Discontinuity

1. Difference in Difference and Regression Discontinuity

2. Review • From Lecture I • Causality is difficult to Show from cross sectional observational studies • What caused what? • X caused Y, Y caused X • Omitted Variable Bias/Confounding • In some cases you can say whether the estimate is an upper-bound or lower bound estimate • Other times impossible to sign bias since omitted variables bias the coefficient of interest positively and negatively. Net effect impossible to determine a-priori.

3. Review (cont.) • Discussed Randomized Control Trials as a simple (but not necessarily practical) way to solve the causality problem • Randomization works because we can be sure about temporal precedence • Randomization works because treatment and control groups are balanced on observables and un-observables

4. Review (cont.) • Also quickly presented some other commonly used research designs • X 01 - Observe only data from post treatment (X) • 01 X 02 – Observe data from pre and post treatment periods • 01 02 X 03 – Observe data from pre and post treatment; observe a longer pre period • Common Feature of all these designs is that there is NO CONTROL GROUP

5. Difference in Difference I • 01 X 02 03 04 • 01 is the pre-period treatment group data • 02 is the post intervention treatment group data • 03 is the pre-period control group data • 04 is the post intervention control group data

6. Difference in Difference I (cont.) • Let’s Compare to 01 X 02 design • No control group, leads to the strong assumption that over time, without an intervention, Dependent variable of interest would not have changed • 01 X 02 03 04 • Diff. in Diff. treatment design accounts for the fact that dependent variable might change even if there were no intervention • Similar to the RCT framework the control group provides the counterfactual

7. Difference in Difference I (cont.) • Simplest representation is 2 X 2 Table • Diff. in Diff. Estimate= • [E(YT1) – E(YT0)] – [E(YC1) – E(YC0)] • Same result even if you calculate = [E(YT1) – E(YC1)] – [E(YT0) – E(CT0)]

8. Difference in Difference I (cont.) • In words: you are subtracting out the change in the control group from the change in the treatment group • If the treatment had not effect then this is tantamount to saying that the two differences are equal • If the treatment had an effect then either the first term is bigger than the second term (positive effect) or the second term is bigger than the first term (negative effect)

9. Difference in Difference II • Why is Diff. and Diff. powerful? • MAIN REASON: We have a control group • Another problem with cross-sectional studies is that we worry about unobserved and hard to measure differences between the treatment and control group • In the difference in difference estimate, Unobserved differences across treatment and control that stay constant over time are differenced out • Another way of saying this is that these unobserved unchanging characteristics effect the Level but not the changes

10. Difference in Difference III • Problems with Difference in Difference Estimation • Lets remember What made RCT powerful • We knew the assignment mechanism: RANDOMIZATION • Note that there is no randomization in Diff. in Diff • Unit of observation (for ex. state) still chooses whether or not to get treatment • Choice leads to the potential problem that treatment and control groups are different • Consequently we are still concerned with some of the usual problems from cross-sectional studies

11. Difference in Difference III (cont.) • Main Concern is “History” • How can we be sure that other interventions are also not simultaneously occurring with treatment? • For ex. Some states in an effort to reduce smoke might enact anti-smoking laws in public spaces • Very possible that the states that enact anti-smoking laws simultaneously enact other anti-smoking measures as well (increase advertising, increase taxes etc.) • For these changes not to bias the diff. in diff. estimate we would have to argue that the control group also enacted these other changes at the same time.

12. Difference in Difference III (cont.) • Specification Checks • Plot pre intervention trends over time for dependent variables for treatment and control groups separately. • IF trends are parallel in treatment and control groups and see sudden change after intervention then you are potentially safe • If trends are not parallel then possible bias from other sources • Create False Treatments and Redo estimation • For ex. If intervention happened in 1990, assign intervention in treatment group to 1989 and see if you still find an effect • If you find an effect then likely that something else is driving your findings

13. Difference in Difference III (cont.) • Use an outcome that shouldn’t be affected by the intervention and redo estimation

14. Difference in Difference IV • Still Other Concerns • Policy intervention is tied to outcome • Difference in Difference will overstate true effect • Mean reversion is again a potential problem • My sense is that this is only a problem for some outcomes (wages is a good ex.) • Long term effect might be difficult to estimate • Estimate is most reliable right after intervention • Long term effects likely confounded by other variables • Functional Form • Means or Logs

15. Card & Krueger - An Example • What is the Effect of a Minimum Wage increase on employment? • Theory says rise in wages should lead to less employment • Firms are profit-maximizing already, taxing one input (labor) should lead to a decrease in it’s use

16. Card & Krueger (cont.) • NJ enacted a state law that increased the minimum wage from 4.25 to 5.05 • Effective April 1, 1992 • Card and Krueger(1994) use a Diff. in Diff. research design to examine whether this change led to lower employment • Control group is Pennsylvania where the minimum wage did not change over this time period

17. Card & Krueger (cont.) • Card & Krueger Look at the effects in Fast Food Industry, Why? • Leading employer of low-wage workers • Easier to measure prices, employment and wages in this industry • Survey Burger King, KFC, Wendy’s and Roy Roger’s chains • Exclude McDonalds because McDonalds had a poor response rates to surveys in previous work • Initial survey conducted in late February and early March 1992, • A month before the NJ minimum wage increase • Secondary Survey conducted in November and December 1992

18. Card & Krueger (cont.) • Around 80% response rate in pre-period • 90% of these 80% responded in post-period • One Key question: Is the wage increase in N.J. meaningful? • Yes, average starting wage in New Jersey restaurants increased by 10% (4.61 to 5.05) • In wave 1: 31% had a starting wage of 4.25 • In Pennsylvania, In wave 1, average starting wage in Pennsylvania was 4.63 and • In wage two there was no change

19. Card & Krueger: Results Avg. Full Time Employees Before Avg. Full Time Employees After • Diff in Diff Estimate: • [21.03-20.44] – [21.17-23.33] • [.59]-[-2.16]=2.75 • Standard Error on estimate is 1.36 • Conclusion: Estimate is positive but not statistically significant at the 5% level NJ PA

20. C&K Results (cont.) • Lets compare to the 01X02 design • Question: Given the C&K data what would you have concluded about the effect of the increase in minimum wage if you used this design? • This simpler design would have said that the effect of the minimum wage hike is positive and the magnitude=.59 • The Diff. in Diff. estimate also says the effect of the minimum wage hike is positive but the magnitude is now 2.7 • Including a control group increases the 01X02 estimate by a factor of close to 5

21. C&K Results (cont.) • Regression Framework • Each observation in the data is a store • Dependent variable is Change in employment • Independent variables include region, chain dummies (burger king etc.) • State Dummy for whether or not in New Jersey • Regression coefficient on State Dummy: 2.33 • On average the law leads to an increase of 2.33 employees • But standard error on the estimate is 1.33 so not statistically different from zero

22. C&K-Other Specifications (cont.) • Some stores not affected if they are already above the minimum wage • Create a “GAP” variable • 0 for stores in Pennsylvania • 0 for stores in NJ whose wage is already above 5.05 • (5.05-initial wage)/initial wage for other NJ states • % increase in wages • Again find a positive effect but not statistically significant

23. C&K-Other Specifications (cont.) • % change in employment in the dependent variable • Exclude management employees • Include part time workers in employment • Exclude stores along the coast of NJ • These stores might have a different seasonal pattern • Finally surveyors called some stores in NJ more often to get data. Exclude these stores from sample • NONE OF THESE CHANGES AFFECT THE BASIC RESULTS

24. C&K - Other Specifications (cont.) • Non Wage-Offsets • Offset raise in minimum wage by reducing non-wage compensation (fringe benefits) • Main fringe benefit is free and reduced price meals • Do not find any changes in this measure • Future wage offsets • Reduce the rate at which salaries increase • Examined the average time to first pay raise

25. Regression Discontinuity • Arbitrary Threshold determines whether or not a unit gets assigned to treatment or Control group • Anti-Discrimination law only applies to firms with at least 15 employees • Rabbinic Scholar Maimonides says Class size cannot exceed 40, if so must group student into smaller classes • For ex. 42 students means average class size is 20.5 • 80 students means two classes of size 40 but 81 students means average class size of 27

26. Regression Discontinuity (cont.) • In this framework, for most examples, being above threshold implies you are in the treatment group • In this framework, for most examples, being below threshold implies you are in the control group • Look for a Change in magnitude of the outcome variable right around this threshold

28. Regression Discontinuity (cont.)- • This research design might make you think of 01 X 02 • But it’s not? Why is that? • For one thing there is no time component • Second 01 is NOT A VALID control group

29. Regression Discontinuity • Two types of regression discontinuity • Sharp Regression Discontinuity • W_I = 1 if X_I >= C • All units with X >= C are assigned the treatment • All units with X< C are assigned to control

30. Probability of Assignment

31. Regression Discontinuity • Sharp Regression Discontinuity • We assume that effect without treatment is linear • There is no way to verify this since treatment is assigned to individuals above the cutoff

32. Potential and Observed Outcome

33. Regression Discontinuity • Fuzzy Regression Discontinuity Design • Probability of receiving does not have to be 1 at the threshold • For ex. Individuals above some threshold could be offered a treatment • The offer does not lead all individuals to take up treatment • As an example think of a voucher scheme that allows people to move neighborhoods. • For some individuals voucher amount is not enough to get them to comply