Prepared for: Agency for Healthcare Research and Quality (AHRQ) www.ahrq.gov

1. Estimation and Reporting of Heterogeneity of Treatment Effects in Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) www.ahrq.gov

2. Outline of Material This presentation will: • Summarize prior knowledge on treatment-effect modifiers and reference sources • Prespecify subgroups to be evaluated • Specify the hypothesized direction of effect within subgroups and the significance levels that will be used to assess statistical significance • Describe how confounding will be addressed • Describe statistical approaches that will be used to test for interactions for prespecified covariates • Describe how overall and subgroup effects will be reported if an interaction test is or is not significant

3. Introduction • Randomized controlled trials (RCTs) and observational comparative effectiveness studies usually report an average treatment effect. • However, the same treatment can affect people differently. • Restriction of RCTs to homogeneous groups improves statistical power. • However, it also contributes to limited generalizability of study results. • Determining real-world effects requires additional information and methods.

4. Heterogeneity of Treatment Effect • Nonrandom variability in the direction or magnitude of a treatment effect • Important for making decisions based on knowing how well a treatment is likely to work for an individual or group of similar individuals • Implications for patients, clinicians, and policymakers • Can be observed across different populations (e.g., children, older adults, patients with comorbidities)

5. Treatment Effect Modification • If two or more exposure variables act in concert to cause disease, the effect of exposure on outcome (treatment effect) differs according to the level of the other factor(s). • Sound inferences require accounting for factors that modify the effect of the exposure. • Perspectives on interaction: • Biological • Statistical • Public health • Individual decisionmaking

6. Goals for Analyzing Heterogeneity ofTreatment Effect • Analyses of heterogeneity of treatment effect have two main goals: • To estimate treatment effects in clinically relevant subgroups (subgroup analysis) • To predict whether an individual might benefit from a treatment (predictive learning)

7. Subgroup Analysis • This type of analysis evaluates the treatment effect for a number of subgroups, one variable at a time. • A test is conducted to evaluate if a subgroup variable has a statistically significant interaction with the treatment indicator. • If the interaction is significant, the treatment effect is estimated separately at each level of the categorical variable. • Cautions for tests of interactions: • Generally have low power to detect subgroup effects • Do not correspond to an assessment of biological interaction

8. Types of Subgroup Analysis

9. Potentially Important Subgroup Variables • Important subgroups are ones for which limited data are typically available. • Subgroup variables must be true covariates (predefined and unaffected by treatment). • Additional important subgroup variable types: • Demographic • Pathophysiologic variables • Comorbidities • Concomitant exposures • Genetic markers

10. General Considerations for Subgroup Analyses • Heterogeneity in observational studies can be due to chance or bias. • Evaluate chance findings by testing for interaction. • Avoid bias by adhering to sound study design. • Observational studies should be designed and analyzed in the same manner as randomized controlled trials to ensure trustworthy results. • Stratification on the propensity to receive a medication or strength of indication can identify clinically relevant subgroups. • Further investigation is warranted if heterogeneity is observed.

11. Conclusion • Observational studies often have broad inclusion/exclusion criteria. • With an increase in generalizability comes the potential for heterogeneity of treatment effects (HTEs). • It is important to understand the potential for HTE before conducting observational comparative effectiveness research. • It is important to ensure adequate statistical power for confirmatory subgroup analyses. • It is necessary to appropriately report the results of an interaction test.

12. Summary Checklist (1 of 2)

13. Summary Checklist (2 of 2)