Poor Research Designs in Policy Impact Studies: “Lies, Damn Lies, and Statistics”

Poor Research Designs in Policy Impact Studies: “Lies, Damn Lies, and Statistics” AHRQ 2007 Annual Conference: Improving Healthcare, Improving Lives September 26, 2007 Stephen Soumerai, ScD Department of Ambulatory Care and Prevention Harvard Medical School and Harvard Pilgrim Health Care

Posttest-Only Design Treatment Group Time Time X O1 X O1 O1 Time O1 X O2 Weak Designs: No Causal InferenceWhat Threats to Validity? Posttest-Only Design with Nonequivalent Groups (cross-sectional) Treatment Group Comparison Group Pretest-Posttest Design Treatment Group

Non-equivalent Control Group Design Time Treatment Group O1X O2 R O1 O2 Comparison Group Time Series Design Time Treatment Group Comparison Series possible O1 O2 O3 X O4 O5O6 O1 O2 O3 O4 O5O6 Strong Quasi-Experimental Designs

Reported Effectiveness of Printed Education Materials Alone in Well-Designed vs. Inadequately Controlled Studies Adequately Controlled Studies Inadequately Controlled Studies Soumerai et al, Milbank Q 1989; 67: 268-317

Example of an Inadequate Cross-sectional Design: Problems with Small Samples and Outliers Effect Attributed by MCOP

Post-only Evaluations of Several Drug Cost-Containment Policies on Medication Use and Health Outcomes Conclusions: Mixed effects on outcomes and costs Design: Post-policy comparisons of several groups (e.g., with and without employer insurance) • No data on baseline comparability • Statistical adjustment for group differences Problems: • Study groups were already different with respect to SES and health status • Instrumental variables, propensity scores, etc. can’t fully control for bias

Use Longitudinal Models • Increases statistical power in quasi-experimental studies • Uses information on trends • Multiple pre- and post-measurements of outcomes • Provide graphical evidence: visible versus statistical

Figure 1: Reductions in benzodiazepine use after Triplicate Prescription Policy among patients living in neighborhoods with different racial compositions Triplicate Prescription Policy Pearson et al, Arch Intern Med 2006; 166:572-9

Benzodiazepine Use and Incidence of Hip Fracture among Women in Medicaid Before and After NY Regulatory Surveillance Bz Use among Female Users before Policy,% Cumulative Incidence of Hip Fracture per 100000 Female Users before Policy

Summary Points • Longitudinal data allow for strong quasi-experimental designs • Provide more valid results • Visible effects almost always significant • Creative use of comparison series • Unexposed comparison population • High risk subgroups • Unintended outcomes

Poor Research Designs in Policy Impact Studies: “Lies, Damn Lies, and Statistics”