Download
1 / 8
80 likes | 175 Views
PolMeth2009: Freedman Panel. Regression Adjustments to Experimental Data: Do David Freedman’s Concerns Apply to Political Science? Donald P. Green Yale University. Using covariates in the analysis of experimental results: the conventional view. Benefit #1: addresses random imbalance
E N D
PolMeth2009: Freedman Panel Regression Adjustments to Experimental Data: Do David Freedman’s Concerns Apply to Political Science? Donald P. Green Yale University
Using covariates in the analysis of experimental results: the conventional view • Benefit #1: addresses random imbalance • Benefit #2: increases precision by reducing disturbance variance • Drawback #1: burns up degrees of freedom • Drawback #2: increases discretion, particularly in the absence of an ex ante analysis plan
Freedman’s critique of covariate adjustment • Doesn’t follow from the experimental design • Asymptotically unbiased but may be severely biased in finite samples • Conventional regression estimates of standard errors may be severely biased
Freedman’s setup • Assign a population of size n to treatment and control groups of size m and n-m, respectively • Potential outcomes model, with responses that are deterministic functions of experimental assignments • When assessing unbiasedness, consider the average estimate across all possible random assignment
Why the fuss? • Experiments are becoming increasingly common, and covariate adjustment using regression is regarded as benign standard operating procedure • Freedman’s claim that finite-sample bias is appreciable for n < 500 encompasses a large proportion of experimental studies published in political science
Aims of my paper • Evaluate the magnitude of the bias for varying n • Simulated data • Real data (from experiments that have been reconfigured so that treatment and control are latent potential outcomes) • Assess when biases are large and whether the symptoms of bias are detectable
Results • Simulated examples: although it is possible to construct examples with severe biases, these tend to involve n<20 and noticeably heterogeneous treatment effects • Analysis of actual experimental data shows very little bias in estimated treatment effects and fairly accurate estimated standard errors
Bottom line • Freedman’s legacy is to challenge unreflective use of off-the-shelf statistical methods • Regression is not unproblematic if applied to small populations with heterogeneous treatment effects, but now we have a clearer idea of what “small” means as a practical matter
