Comparing Results from RCTs and Quasi-Experiments that share the same Intervention Group

1. Comparing Results from RCTs and Quasi-Experiments that share the same Intervention Group Thomas D. Cook Northwestern University

2. Why RCTs are to be preferred • Statistical theory re expectations • Relative advantage over other bias-free methods--e.g., regression-discontinuity (RDD) and instrumental variables (IV) • Ad hoc theory and research on implementation • Privileged credibility in science and policy • Claim that non-exp. alternatives routinely fail to produce similar causal estimates

3. Dissimilar Estimates • Come from empirical studies comparing exp. and non-exp. results on same topic • Strongest are within-study comparisons • These take an experiment, throw out the control group, and substitute a non-equivalent comparison group • Given the intervention group is a constant, this is a test of the different control groups

4. Within-Study Comparison Lit. • 20 studies, mostly in job training. Of the 14 in job training reviews contend: • (1) no study produces a clearly similar causal estimate, including Deheija & Wahba • (2) Some design and analysis features associated with less bias, but still bias • (3) the average of the experiments is not different from the average of the non-experiments--but be careful here and note the variance of the effect sizes differs by design type

5. Brief History of Literature on Within Study Comparisons • LaLonde; Fraker & Maynard • 12 subsequent studies in job training • Extension to examples in education in USA and social welfare in Mexico, never yet reviewed

6. Policy Consequences • Department of Labor, as early as 1985 • Health and Human Services, job training and beyond • National Academy of Sciences • Institute of Educational Sciences • Do within-study comparisons deserve all this?

7. We will: • Deconstruct „non-experiment“ and compare experimental estimates to • 1. Regression-discontinuity estimates • 2. Estimates from difference-of-differences (fixed effects) design • Ask: Is general conclusion about the inadequacy of non-experiments true across at least these different kinds of non-experiment

8. Criteria of Good Within-Study Comparison Design 1. Variation in mode of assignment--random or not 2. No third variables correlated with both assignment and outcome--e.g., measurement 3. Randomized experiment properly executed 4. Quasi-experiment good instance of “type” 5. Both design types estimate the same causal entity--e.g, LATE in regression-discontinuity 6. Acceptable criteria of correspondence between design types--ESs seem similar; not formally differ; stat significance patterns not differ, etc.

9. Experiments vs. Regression-Discontinuity Design Studies

10. Three Known within-Study Comparisons of Exp and R-D • Aiken, West et al (1998)- R-D study; experiment; LATE; analysis; results • Buddelmeyer & Skoufias (2003)-R-D study; experiment; LATE; analysis; results • Black, Galdo & Smith (2005)-R-D study; experiment; LATE; analysis; results

11. Comments on R-D vs Exp. • Cumulative correspondence demonstrated over three cases • Is this theoretically trivial, though? • Is it pragmatically significant, given variation in implementation in both the experiment and R-D? • As “existence proof”, it belies over-generalized argument that non-experiments don’t work • As practical issue, does it mean we should support RDD when treatments are assigned by need, merit. • Emboldens to deconstruct non-experiment further

12. Experiment vs Differences-in-Differences • Most frequent non-experimental design by far across many fields of study • Also modal in within-study comparisons in job training, and so it provides major basis for past opinion that non-experiments are routinely biased • We review: 3 studies with comparable estimates • 14 job training studies with dissimilar estimates • 2 education examples with dissimilar estimates

13. Bloom et al • Bloom et al (2002; 2005)--job training the topic • Experiment 11 sites - 8 pre earning waves; 20 post • Non-Experiment = 5 within-state comparisons; 4 within-city; all comparison Ss enrolled in welfare • We present only control/comparison contrast because treatment time series is a constant

14. Issue is: • Is there overall difference between control groups randomly or non-randomly formed? • If yes, can statistical controls—OLS, IV (incl. Heckman models), propensity scores, random growth models—eliminate this difference? • Tested 1O modes, but only one longitudinal • Why we treat this as d-in-d rather than ITS

15. Bloom et al. Results

16. Bloom et al. Results (continued)

17. Implications of Bloom et al • Averaging across the 4 within-city sites showed no difference-also true if 5th between-city site added • Selecting within-study comparisons obviated the need for statistical adjustments for non-equivalence--design alone did it. • Bloom et al tested differential effects of statistical adjustments in between-state comparisons where there were large differences • None worked, or did better than OLS

18. Aiken et al (1998) Revisited The experiment. Remember that sample was selected on narrow range of test score values • Quasi-Experiment--sample selection limited to students who register late or cannot be found in summer but who score in the same range as the experiment • No differences between experiment and non-experiment on test scores or pretest writing tests • Measurement identical in experiment and non-exp

19. Results for Aiken et al • Writing standardized test = .59 and .57 - sig • Rated essay = .06 and .16 – ns • High degree of comparability in statistical test results and effect size estimates

20. Implications of Aiken et al • Like Bloom et al, careful selection of sample gets close correspondence on important observables. • Little need for stat adjustment for non-equivalence limited only to unobservables • Statistical adjustment minor compared to use of sampling design to construct initial correspondence

21. What happens if there is an initial selection difference? • Shadish, Luellen & Clark (2006)

22. Figure 1: Design of Shadish et al. (2006) N = 445 Undergraduate Psychology Students Pretests, and then Random Assignment to Randomized Experiment n = 235 Randomly Assigned to Nonrandomized Experiment n = 210 Self-Selected into Mathematics Training n = 79 Vocabulary Training n = 131 Mathematics Training n = 119 Vocabulary Training n = 116 All participants measured on both mathematics and vocabulary outcomes

23. What’s special in Shadish et al • Variation in mode of assignment • Hold constant most other factors thru first RA--population/measures /activity patterns • Good experiment? Pretests; short-term and attrition; no chance for contamination. • Good quasi-experiment? - selection process; quality of measurement; analysis and role of Rosenbaum

24. Results Shadish et al

25. Implications of Shadish et al • Here the sampling design produced non- equivalent groups on observables, unlike Bloom • Here the statistical adjustments worked when computed as propensity scores • However, big overlap in experimental and non-experimental scores due to first stage random assignment, making propensity scores more valid • Extensive, unusually valid measurement of a relatively simple selection process, though not homogeneous.

26. Limitations to Shadish et al • What about more complex settings? • What about more complex selection processes? • What about OLS and other analyses? • This is not a unique test of propensity scores!

27. Examine Within-Study Comparison Studies with different Results • The Bulk of the Job Training Comparisons • Two Examples from Education

28. Earliest Job Training Studies: Adding to Smith/Todd Critique • Mode of Assignment clearly varied • We assume RCT implemented reasonably well • But third variable irrelevancies were not controlled, esp location and measurement, given dependence on matching from extant data sets • Large initial differences between randomly and non-randomly formed comparison groups • Reliance on statistical adjustment to reduce selection, and not initial design

29. Recent Educational Examples

30. Agodini & M. Dynarski (2004) • Drop-out prevention experiment, 16 m/h schools • Individual students, likely dropouts, were randomly assigned within schools—16 replicates • Quasi-Experiment—students matched from 2 quite different sources: middle school controls in another study, and national NELS data. • Matching on individual and school demographic factors • 4 outcomes examined and so in non-experiment • 128 propensity scores -16 x 4 x 2--computed basically from demographic background variables

31. Results • Only 29 of 128 cases were balanced matches obtained • Why quality matching so rare? In non-experiment, groups hardly overlap. Treatment group is high and middle schools, but comparisons are middle only or from a very non-local national data set • Mixed pattern of outcome correspondences in 29 cases of computable propensity scores. Not good • OLS did as well as propensity scores

32. Critique • Who would design a quasi-experiment this way? Is a mediocre non-experiment being compared to a good experiment? • Alternative design might have been: • 1. Regression-discontinuity. • 2. Local comparison schools, same selection mechanism to select similar comparison students. 3 Use of multi-year prior achievement data.

33. Wilde & Hollister (2005) • The Experiment—reducing class size in 11 sites; no pretest used at the individual level • Quasi-experimental design—individuals in reduced classes matched to individual cases from other 10 sites • Propensity scores; mostly demographic • Analysis treat each site as a separate experiment • And so 11 replicates comparing an experimental and non-experimental effect size

34. Results • Low level of correspondence in experimental and non-experimental effect sizes across the 11 sites • So for each site it makes a causal difference whether experiment or quasi-experiment • When aggregated across sites, results closer: exp = .68; non-exp = 1.07 • But they do reliably differ

35. Critique • Who would design a quasi-exp on this topic without a pretest on same scale as outcome? • Who would design it with these controls? • Instead select controls from one or more matched schools on prior achievement history • Again, a good experiment is being compared to a bad quasi-experiment • Who would treat this as 11 separate experiments vs. a more stable pooled experiment? Even the authors, pooled results are much more congruent.

36. Hypothesis is that... • The job training and educational examples that produce different conclusions from the experiment are examples of poor quasi-experimental design • To compare good exp to poor quasi-exp is to confound a design type and the quality of its implementation—a logical fallacy • But I reach this conclusion ex post facto and knowing the randomized experimental results in advance

37. Big Conclusions: • R-D has given results not much different from experiment in three of three cases. • Simpler Quasi-Experiments tend to give same results as experiment if: (a) population matching in the sampling design—Bloom and Aiken studies, or if (b) careful conceptualization and measurement of selection model, as in Shadish et.

38. What I am not Concluding: • That well designed quasi-experiment is as good as an experiment. Difference in: • Number and transparency of assumptions • Statistical power • Knowledge of implementation • Social and political acceptance • If you have the option, do an experiment because you can rarely put right by statistics what you have messed up by design

39. What I am suggesting you consider: • Whether this be a unit on RCTs or quality causal studies • Whether you want to do RDD studies in cases where an experiment is not possible because resources are distributed otherwise • Whether you want to do quasi-experiments if group matching on the pretest is possible, as in many school-level interventions?

40. More Contentiously if: • The selection process can be conceptualized, observed and measured very well. • An abbreviated ITS analysis is possible, as in Bloom et al. • The instinct to avoid quasi-experiments is correct, but it reduces the scope of the causal issues that can be examined

43. Shadish, Luellen & Clark (2006)

44. Shadish, Luellen & Clark (2006)

45. Results-Aiken et al • pretest values on SAT/CAT, 2 writing measures • Measurement framework the same • Pretest ACTs and writing - ns exp vs non • OLS tests • Results for writing test = .59 and .57 - sig • Results for essay = .06 and .16 - ns

46. Bloom et al Revisited • Analysis at the individual level • Within city, within welfare to work center, same measurement design • Absolute bias- yes • Average bias none across 5 within-state sites, even w/o stat tests • Average bias limited to small site and non-within-city site-Detroit vs Grand Rapids

47. Correspondence Criteria • Random error and no exact agreement • Shared stat sig pattern from zero - 68% • Two ESs not statistically different • “Comparable” magnitude estimates • One as percent of other • Indulgence, common sense and mix

48. Our Research Issues • Deconstructing “non-experiment”--do experimental and non-experimental ESs correspond differently for R-D, for ITS, and for simple non-equivalent designs? • How far can we generalize results about invalidity of non-experiments beyond job training? • Do these within-study comparison studies bear the weight ascribed to them in evaluation policy at DoL and IES?