Evaluation of public policies: alternative methods. Second week . Topic 1. Intro and Diff in Diff

1. Evaluation of public policies:alternative methods.Second week . Topic 1.Intro and Diff in Diff Orazio Attanasio and Marcos Vera-Hernandez Los Andes July 14- July 25 2008

2. Outline • Step back to look at the evaluation problem and state where we stand. • Provide basic notation again. • Difference in Difference Blundell, R. and M. Costa-Diaz: “Alternative Approaches to Evaluation In Empirical Microeconomics” December 2007. http://www.ucl.ac.uk/~uctp39a/Blundell-CostaDias-Dec-2007.pdf

3. Outline • Regression Discontinuity Design • Instrumental Variables • Control Functions • Structural Models for Policy Evaluation

4. The basic evaluation problem • We want to establish how a specific outcome on an individual who receives or is exposed to a public policy is affected by the policy. • The ‘policy’ can be intended quite widely • For example estimating the return to education can be seen as an evaluation problem. • The basic difficulty in establishing the counterfactual: • What the outcome would have been in the absence of a policy

5. The fundamental evaluation problem: some basic notation

6. The fundamental evaluation problem: some basic notation

7. The basic evaluation problem • The ‘counterfactual problem’ cannot be solved at the individual level. • Randomization is classic solution. • This would allow us to estimate ‘average treatment effects’. • Or, related parameters, (ATT).

8. Estimation of treatment effects • Different alternatives differ in the amount of structure one wants to put into the problem. • Randomization: • no structure or special assumptions (almost) • No ability to extrapolate. • Matching: • Selection on observable • Allow for heterogeneous treatment effects • No unobservable differences between treatment and control • Difference in Difference: • allow for unobservable but constant through time

9. Estimation of treatment effects • Instrumental variables: • Model participation to correct selection bias • Homogeneous effects? • Control functions • Model participation • Allow more naturally for heterogeneous effects, • Allow more easily for non linear models • Structural models • Strong assumptions • Allow extrapolation and simulations of program changes

10. The basic evaluation problem: additional notation

11. The basic evaluation problem: additional notation

12. The basic evaluation problem • The ‘counterfactual problem’ is common to many sciences, not only social sciences. • Why is it particularly difficult in social sciences? • Randomization is seldom available • There are good reason to believe that in many situations there is correlation between u and d

13. The basic evaluation problem

14. Homogeneous v heterogeneous treatment • Homogeneous effects

15. Homogeneous v heterogeneous treatment • Heterogeneous effects

16. Difference in difference and natural experiments • Natural experiments are changes in legislation or other historical accidents that can be interpreted as ‘random’. • That is they can be interpreted as affecting participation without affecting outcomes. • They can therefore be used to create ‘control’ groups.

17. Changes over time • Treated individuals observed before the program can constitute a ‘control’ for themselves. • We can interpret before /after comparison as identifying treatment effects if there is nothing that changes the outcome variable except the program. • Of course this is a very strong and untestable assumption.

18. Difference in difference • But if we have one group affected by the program and one that is not and we observed them twice, once before and once after the program: • We can use the pre-program differences to estimate permanent differences between the groups. • The comparison after the program can then be ‘corrected’ to identify the effect of the program • Notice that the diff in diff estimator can be obtained both when we have longitudinal data and when we have repeated cross sections from representative groups

19. Difference in difference

20. Difference in difference Assume:

21. Difference in difference

22. Difference in difference

23. Difference in difference

24. Difference in difference • Notice that the previous equation can be estimated by OLS consistently on longitudinal data. • However, the same equation can be estimated from repeated cross sections.

25. Problems with DiD • Ashenfelter’s dip. • The procedure does not control for temporary shocks that affect participation. • This can seriously bias the procedure.

26. Problems with DiD:Different macro trends. • It is possible that the two groups are affected differently by time trends • This would bias the estimator (see graphical example) • We can check for this with pre-program data. • We could also ‘correct’, under some assumptions, for this bias.

27. Problems with DID • Compositional changes when longitudinal data are not available

28. Non linear DID • The DiD idea relies on additive components to get the outcome variable. • This assumption can be unrealistic, especially in the case of dummy variables.

29. Non linear DID oitfollows a distribution F with inverse F-1

30. Non linear DID • For simplicity assume the absence of heterogeneous fixed effects • But still maintain heterogeneous impacts) • Under normality (F is the normal distribution), the model becomes a standard probit • The temptation to obtain the diff in diff is to run a simple probit on the pooled data (t=0 and t=1), with time dummies, group dummies and interaction. • Could the interaction coefficient be interpreted as the impact as in the linear (OLS) case? • NO!!!!

31. Non linear DID Require distributional assumptions on this term You cannot do standard probit

32. Non linear DID

33. Non linear DID

34. Non linear DID

35. Additional tricks • Combining matching and DID • Could DiD be the right thing to do in the case of a randomized experiment? • Would one to controls to a randomized experiment?

36. Combining matching and DiD • One could allow the constant β to be a function of observables. • One could also assume that the effects are a function of these observables. • Therefore one could use matching techniques on first differenced data • (if longitudinal data are available) • Or use repeated cross sections.

37. Combining matching and DiD

38. Would DiD the right thing to do in a randomized trial? • Yes if it turns out that the randomization sample is small. • It could be that the treatment and control samples are imbalanced out of bad luck • What about efficiency arguments: • It depends. It could go either way.

39. Example of diff in diff evaluation:The evaluation of Familias en Accion • Familias en Accion is a CCT program started in Colombia in 2002 with a loan from the World Bank and the IADB • The evaluation was one of the conditions imposed by the loan • The antecedent was PROGRESA in Mexico, where the expansion phase of the program was used to build an evaluation based on the randomized allocation of the program across communities.

40. Example of diff in diff evaluation:The evaluation of Familias en Accion • The expansion phase in Colombia was also quite long. • However, for political reasons it was not possible to randomize across communities. • It was decided to use a treatment/control comparison and use diff in diff combined with matching techniques.

41. Example of diff in diff evaluation:The evaluation of Familias en Accion • The treatment municipios were chosen by the government. • Treatment list (627 municipalities): • Municipios with less than 100k (no capitals), • with enough infrastructure, • with complete sisben registry as of Dec. 1999, • With a bank! • Treatment sample: • Representative list of municipalities • Stratified by region (5 regions) and by level of infrastructure (5) • Control sample: • Municipalities in the same strata used to stratify the treatment sample that were ‘similar’ to the treatment municipalities actually in the sample.

42. Example of diff in diff evaluation:The evaluation of Familias en Accion • The government became very anxious to start the program quickly in 2002 (approaching elections? End of mandate effects? Crisis? ) • In early 2002 became clear that it was difficult to collect a baseline before the start of the program in the treatment municipalities. • Two decisions: • Strong negotiations with the government (intermediated by the WB and the IADB) to prevent the start of the program in at least some municipalities • Introduction in the questionnaire of retrospective questions. • Possible problems: • Low power (reduction in sample size) • Anticipation effects,

43. Example of diff in diff evaluation:The evaluation of Familias en Accion • For the evaluation we collected three surveys. • Baseline 2002 (but some treatment were already receiving payments) • First Follow up 2003 (attrition 6%) • Second follow up 2005/6 (attrition 10%) • Very long household survey (11,500 hh) complemented with surveys of schools, hospitals, health centres, localities, HC’s.

44. Difference in difference Y(i,t) = e D + bt + gt + u

45. Ejemplo 1: gT =gC: e=YT2-YC2

46. Ejemplo 2: gT ≠ gC: YT2-YC2= e+gT-gC YT1-YC1 = gT-gC e= (YT2-YC2 )– (YT1-YC1)

47. Extensiones necesarias • El programa empezó a operar en algunos municipios tratamiento antes de la línea de base: • Aunque esto permitió la precoz evaluación de resultados de impacto, ahora da lugar a problemas metodológicos y de eficiencia. • Diferencia en intensidad de la intervención.

48. TCP y TSP • Tenemos TCP y TSP • Para unas variables tenemos frecuencias pre – línea de base

49. YT1=b0 + gT + u YP0=b0 + gT + u YC0=b0 + gC + u YT1=b1 + gT + u YP1=e+b1 + gT + u YC1=b1 + gC + u YC2=b2 + gC + u YP2=e+b2 + gT + u YT2=e+b2 + gT + u TCP y TSP (cont)

50. TSP y TCP (cont) e=(YT2-YC2 )– (YT1-YC1) e=(YT2-YP2 )– (YT1-YP1) Estas estimaciones pueden ser combinadas para incrementar la eficiencia