Estimating Causal Effects with Experimental Data

1. Estimating Causal Effects with Experimental Data

2. Some Basic Terminology • Start with example where X is binary (though simple to generalize): • X=0 is control group • X=1 is treatment group • Causal effect sometimes called treatment effect • Randomization implies everyone has same probability of treatment

3. Why is Randomization Good? • If X allocated at random then know that X is independent of all pre-treatment variables in whole wide world • an amazing claim but true. • Implies there cannot be a problem of omitted variables, reverse causality etc • On average, only reason for difference between treatment and control group is different receipt of treatment

4. Why is this useful?An Example: Racial Discrimination • Black men earn less than white men in US LOGWAGE | Coef. Std. Err. t -----------+------------------------------- BLACK | -.1673813 .0066708 -25.09 NO_HS | -.2138331 .0077192 -27.70 SOMECOLL | .1104148 .0049139 22.47 COLLEGE | .4660205 .0048839 95.42 AGE | .0704488 .0008552 82.38 AGESQUARED | -.0007227 .0000101 -71.41 _cons | 1.088116 .0172715 63.00 • Could be discrimination or other factors unobserved by the researcher but observed by the employer? • hard to fully resolve with non-experimental data

5. An Experimental Design • Bertrand/Mullainathan “Are Emily and Greg More Employable Than Lakisha and Jamal”, American Economic Review, 2004 • Create fake CVs and send replies to job adverts • Allocate names at random to CVs – some given ‘black-sounding’ names, others ‘white-sounding’

6. Outcome variable is call-back rates • Interpretation – not direct measure of racial discrimination, just effect of having a ‘black-sounding’ name – may have other connotations. • But name uncorrelated by construction with other material on CV

7. The Treatment Effect • Want estimate of:

8. Estimating Treatment Effects: the Statistics Course Approach • Take mean of outcome variable in treatment group • Take mean of outcome variable in control group • Take difference between the two • No problems but: • Does not generalize to where X is not binary • Does not directly compute standard errors

9. Estimating Treatment Effects: A Regression Approach • Run regression: yi=β0+β1Xi+εi • Proposition 2.2 The OLS estimator of β1 is an unbiased estimator of the causal effect of X on y: • Proof: Many ways to prove this but simplest way is perhaps: • Proposition 1.1 says OLS estimates E(y|X) • E(y|X=0)= β0 so OLS estimate of intercept is consistent estimate of E(y│X=0) • E(y|X=1)= β0+β1so β1 is consistent estimate of E(y│X=1) -E(y│X=0) • Hence can read off estimate of treatment effect from coefficient on X • Approach easily generalizes to where X is not binary • Also gives estimate of standard error

10. Computing Standard Errors • Unless told otherwise regression package will compute standard errors assuming errors are homoskedastic i.e. • Even if only interested in effect of treatment on mean X may affect other aspects of distribution e.g. variance • This will cause heteroskedasticity • Heteroskedasticity does not make OLS regression coefficients inconsistent but does make OLS standard errors inconsistent

11. ‘Robust’ Standard Errors • Also called: • Huber standard errors • White standard errors • Heteroskedastic-consistent standard errors • Simple to use in practice e.g. in STATA: . reg y x, robust • Statistics course approach • Get variance of estimate of mean of treatment and control group • Sum to give estimate of variance of difference in means

12. Bertrand/Mullainathan:Basic Results

13. Summary So Far • Econometrics very easy if all data comes from randomized controlled experiment • Just need to collect data on treatment/control and outcome variables • Just need to compare means of outcomes of treatment and control groups • Is data on other variables of any use at all? • Not necessary but useful

14. Including Other Regressors • Can get consistent estimate of treatment effect without worrying about other variables • Reason is that randomization ensures no problem of omitted variables bias • But there are reasons to include other regressors: • Improved efficiency • Check for randomization • Improve randomization • Control for conditional randomization • Heterogeneity in treatment effects

15. The Uses of Other Regressors I: Improved Efficiency • Don’t just want consistent estimate of causal effect – also want low standard error (or high precision or efficiency). • Standard formula for standard error of OLS estimate of βis σ2/Var(X) • σ2 comes from variance of residual in regression – (1-R2)* Var(y) • Include more variables and R2 rises – formal proof (Proposition 2.4) a bit more complicated but this is basic idea.

16. The Uses of Other Regressors II: Check for Randomization • Randomization can go wrong • Poor implementation of research design • Bad luck • If randomization done well then W should be independent of X – this is testable: • Test for differences in W in treatment/control groups • Probit model for X on W

17. The Uses of Other Regressors III:Improve Randomization • Can also use W at stage of assigning treatment • Can guarantee that in your sample X and W are independent instead of it being just probabiliistic • This is what Bertrand/Mullainathan do when assigning names to CVs

18. The Uses of Other Regressors IV:Adjust for Conditional Randomization • This is case where must include W to get consistent estimates of treatment effects • Conditional randomization is where probability of treatment is different for people with different values of W, but random conditional on W • Why have conditional randomization? • May have no choice • May want to do it (c.f. stratification)

19. An Example: Project STAR • Allocation of students to classes is random within schools • But small number of classes per school • This leads to following relationship between probability of treatment and number of kids in school:

20. Controlling for Conditional Randomization • X can know be correlated with W • But, conditional on W, X independent of other factors • But must get functional form of relationship between y and W correct – matching procedures • This is not the case with (unconditional) randomization – see class exercize

21. Heterogeneity in Treatment Effects • So far have assumed causal (treatment) effect the same for everyone • No good reason to believe this • Start with case of no other regressors: yi=β0+β1iXi+εi • Random assignment implies X independent of β1i • Sometimes called random coefficients model

22. What treatment effect to estimate? • Would like to estimate causal effect for everyone – this is not possible – Holland’s fundamental problem of statistical inference • Can only hope to estimate some average • Average treatment effect: • Proposition 2.5: OLS estimates ATE

23. Observable Heterogeneity • Full outcomes notation: • Outcome if in control group: y0i=γ0’Wi+u0i • Outcome if in treatment group: y1i=γ1’Wi+u1i • Treatment effect is (y1i-y0i) and can be written as: (y1i-y0i )=(γ1- γ0 )’Wi+u1i-u0i • Note treatment effect has observable and unobservable component • Can estimate as: • Two separate equations • One single equation

24. Combining treatment and control groups into single regression • We can write: • Combining outcomes equations leads to: • Regression includes W and interactions of W with X – these are observable part of treatment effect • Note: error likely to be heteroskedastic

25. Bertrand/Mullainathan • Different treatment effect for high and low quality CVs:

26. Units of Measurement • Causal effect measured in units of ‘experiment’ – not very helpful • Often want to convert causal effects to more meaningful units e.g. in Project STAR what is effect of reducing class size by one child

27. Simple estimator of this would be: • where S is class size • Takes the treatment effect on outcome variable and divides by treatment effect on class size • Not hard to compute but how to get standard error?

28. IV Can Do the Job • Can’t run regression of y on S – S influenced by factors other than treatment status • But X is: • Correlated with S • Uncorrelated with unobserved stuff (because of randomization) • Hence X can be used as an instrument for S • IV estimator has form (just-identified case):

29. The Wald Estimator • This will give estimate of standard error of treatment effect • Where instrument is binary and no other regressors included the IV estimate of slope coefficient can be shown to be:

30. Partial Compliance • So far: • in control group implies no treatment • In treatment group implies get treatment • Often things are not as clean as this • Treatment is an opportunity • Close substitutes available to those in control group • Implementation not perfect e.g. pushy parents

31. An Example: Moving to Opportunity • Designed to investigate the impact of living in bad neighbourhoods on outcomes • Gave some residents of public housing projects chance to move out • Two treatments: • Voucher for private rental housing • Voucher for private rental housing restricted for use in ‘good’ neighbourhoods • No-one forced to move so imperfect compliance – 60% and 40% did use it

32. Some Terminology • Z denotes whether in control or treatment group – ‘intention-to-treat’ • X denotes whether actually get treatment • With perfect compliance: • Pr(X=1│Z=1)=1 • Pr(X=1│Z=0)=0 • With imperfect compliance: 1>Pr(X=1│Z=1)>Pr(X=1│Z=0)>0

33. What Do We Want to Estimate? • ‘Intention-to-Treat’: ITT=E(y|Z=1)-E(y|Z=0) • This can be estimated in usual way • Treatment Effect on Treated

34. Estimating TOT • Can’t use simple regression of y on Z • But should recognize TOT as Wald estimator • Can estimated by regressing y on X using Z as instrument • Relationship between TOT and ITT:

35. Most Important Results from MTO • No effects on adult economic outcomes • Improvements in adult mental health • Beneficial outcomes for teenage girls • Adverse outcomes for teenage boys

36. Sample results from MTO • TOT approximately twice the size of ITT • Consistent with 50% use of vouchers

37. IV with Heterogeneous Treatment Effects (More Difficult) • If treatment effect same for everyone then TOT recovers this (obvious) • But what if treatment effect heterogeneous? • No simple answer to this question • Suppose model for treatment effect is:

38. Proposition 2.6The IV estimate for the heterogeneous treatment case is a consistent estimate of:where:the difference in the probability of treatment for individual i when in treatment and control group

39. Interpretation • This is weighted average of treatment effects • ‘weights’ will vary with instrument – contrast with heterogeneous treatment case • Some cases in which can interpret IV estimate as ATE

40. How will IV estimate differ from ATE • IV is ATE if no correlation between β1i and πi • Previous formula says depends on covariance of β1i and πi • In some situations can sign – but not always • Example 1: no-one gets treatment in the absence of the programme so • If those who get treatment when in the treatment group are those with the highest returns then: • IV>ATE

41. Example 2: treatment is voluntary for those in the control group but compulsory for those in the treatment group • This implies • If those who get treatment in control are those with highest returns then: • IV<ATE

42. Angrist/Imbens Monotonicity Assumption • Case where IV estimate is not ATE • Assume that everyone moved in same direction by treatment – monotonicity assumption • Then can show that IV is average of treatment effect for those whose behaviour changed by being in treatment group • They call this the Local Average Treatment Effect (LATE)

43. Problems with Experiments • Expense • Ethical Issues • Threats to Internal Validity • Failure to follow experiment • Experimental effects (Hawthorne effects) • Threats to External Validity • Non-representative programme • Non-representative sample • Scale effects

44. Conclusions on Experiments • Are ‘gold standard’ of empirical research • Are becoming more common • Not enough of them to keep us busy • Study of non-experimental data can deliver useful knowledge • Some issues similar, others different