Internal Validity

2. Internal Validity • Major threats to internal validity • Selection • Differential maturation • Differential history • Contamination • Strategies to protect internal validity • Randomization • A priori matching or stratification of groups • Objective measures • Independent evaluation personnel who are blind to conditions • Analytic strategies • Lack of randomization only increases the threat potential

3. Validity Of The Analysis • Threats to the validity of the analysis • Misspecification of the analysis model • Low power • Strategies to protect the validity of the analysis • Plan the analysis concurrent with the design • Anticipate all sources of random variation • Anticipate patterns of over-time correlation • Employ strong interventions with good reach • Employ more and smaller groups • Employ analytic strategies to reduce variation • Random assignment of enough groups to each condition is a critical step to ensure the validity of the analysis

4. External Validity • Threats to external validity • Non-representative groups • Non-representative members • Strategies to protect external validity • Recruit a representative sample of groups • Recruit a representative sample of members • Tradeoffs • Natural experiments provide no control over selection of the intervention groups or members. • GRTs allow the investigator complete control to select a representative sample of groups and members. • Quasi-experiments provide some control.

5. Other Considerations • Timing and opportunity • Natural and quasi-experiments require the right opportunity. • Natural experiments operate on their own timeline. • GRTs allow the investigators to create their own opportunities and timeline. • Measurement constraints • Natural and quasi-experiments don't always allow the investigator to impose the desired measurement protocol. • GRTs do not have this constraint. • Sample size • Other factors constant, the sample size requirements are the same for GRTs, natural experiments, and quasi-experiments.

6. Other Considerations • Cost • Measurement costs are likely the same. • No intervention costs in a natural experiment • Intervention costs may range from 20-50% of the direct costs. • Potential effect size • GRTs are sized around an expected effect size. • Natural experiments may have a larger expected effect size, but they may not be sized around that effect size. • Quasi-experiments can be sized around an expected effect size.

7. How Many Units Of Assignment? • GRT • There are usually many of them. • They are randomly assigned to conditions. • Natural experiment • There may be only one unit, made up of multiple subunits. • Example: one school district decides to change its food service operation affecting several middle or high schools. • It would be a very fortunate circumstance to find several schools or districts independently deciding to make a similar change in their food service at the same time. • There is no random assignment. • Quasi-experiment • Possibly multiple units, but not random assignment.

8. Studies With A Single Unit Of Assignment • Studies based on only one group per condition cannot estimate variation due to the group independent of variation due to condition. • Such studies must rely on one of several strong assumptions in their analysis: • Post hoc correction: external estimate is valid • Subgroup or batch analysis: subgroup captures group variance • Fixed-effects analysis: group variance is zero • The first assumption is untestable. • Varnell et al. (2001) evaluated the second and third strategies and found that they are likely to have an inflated Type I error rate in conditions common to GRTs. • These concerns are particularly relevant for natural experiments.

9. Studies With Few Units Of Assignment • For most outcomes considered in health promotion and disease prevention studies, 8-10 groups are needed per arm to allow for the expected intraclass correlation. • Other factors constant, studies with just a few units of assignment will usually be underpowered when analyzed with methods widely recommended for nested designs. • This concern is particularly relevant for natural and quasi-experiments. • Work is underway that would allow investigators to use information from previous studies to improve the power of their small studies, but this work has not yet been vetted by the scientific community.

10. Summary • Group-randomized trials • Randomization provides the statistical basis for the assumption of independence at the group level. • With proper randomization and enough groups… • Potential sources of bias are fairly distributed across the study conditions. • Inferences based on a valid analysis can be as strong as those obtained from a randomized clinical trial. • For this reason, the GRT is the gold standard for design in public health and medicine when allocation of identifiable groups is necessary. • Natural and quasi-experiments • These studies face similar challenges, but do not have randomization or always even multiple units to protect them. • We cannot ignore the design and analytic problems of nested designs just because we don't have a randomized experiment.