Childhood Obesity Scenario: Quasi-Experiments and Natural Experiments Versus RCTs

2. Childhood Obesity Scenario: Quasi-Experiments and Natural Experiments Versus RCTs Steven Gortmaker, Ph.D. Harvard School of Public Health /Harvard Prevention Research Center

3. Definitions • Quasi-experiments: can have all the attributes of randomized controlled trial with pretest and posttest data; key difference: no random assignment to intervention versus control • Natural experiment: can have all the attributes of randomized controlled trial with pretest and posttest data; key differences: • no random assignment • experimenter does not control intervention

4. The Research Question • Can be same in Quasi-experiment and Natural experiment as in Randomized Controlled Trial • However, random assignment can set limits: • Long lead time for the study/results; may preclude rapid evaluations of innovations • Funding agency often pays for the intervention - may preclude evaluations of expensive or complex interventions or policy changes

5. Quasi/Natural Experiment Quasi-experiment has similar data requirements to RCT Natural experiments may need to rely on surveillance data Group Randomized Trial Want pre-post data on key measures Evaluation Data

6. Quasi/Natural Experiment Success dependent on selection of control sample (e.g. propensity matching) Multiple pre-intervention and post-intervention data points can strengthen design Group Randomized Trial The major strength of RCT’s Multiple pre-intervention and post-intervention data points can strengthen design Internal Validity: Control of Selection/Confounding

7. Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Single Data Point Pre and Post-Intervention Control Mean BMI Intervention Intervention Begins Year

8. Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Multiple Data Points Pre and Post-Intervention Control Mean BMI Intervention Intervention Begins Year

9. Quasi/Natural Experiment Similar concerns to RCTS Can study predictors of loss, model loss Natural experiment: a concern is lack of detailed baseline data Group Randomized Trial As with Quasi-experiment, can study predictors of loss, model loss Loss to Follow-up

10. Quasi/Natural Experiment Quasi-experiment: similar to RCT but may have broader sample of participant sites Natural experiment: concerns re selection of intervention sites Group Randomized Trial Participating sites may differ substantially from target population As with Quasi-experiment, can study participant sites versus non-participants External Validity: Generalizability

11. Quasi/Natural Experiment Quasi-experiment: can be similar to RCT, or can capitalize on other funding Natural experiment: intervention costs generally not borne by funding agency Group Randomized Trial Intervention costs often borne by funding agency; can limit cost of intervention program If costs borne by outside agency, difficulty with randomization (but not always!) Intervention Costs

12. Quasi/Natural Experiment Quasi-experiment: can be similar to RCT re data collection Natural experiment: can be less expensive by using extant data Group Randomized Trial Similar data collection costs re Quasi-experiment. A hypothesis: typically data collection costs in RCTs are much larger than intervention costs Evaluation Costs

13. Quasi/Natural Experiment Worse internal validity; but more data points can help Some limited generalizability - particularly in natural experiments Potential to study more innovative, expensive, difficult to implement programs/policies Group Randomized Trial Better internal validity Often limited generalizabilty; but potential to improve More limited programs/policies to study with RCT due to costs and difficulty of randomizing Summary of Trade-Offs