Multivariate Meta-analysis: Notes on Correlations

1. Multivariate Meta-analysis: Notes on Correlations Robert Platt Department of Epidemiology & Biostatistics McGill University Jack Ishak United BioSource

2. Outline • Background – Multivariate MAs • Motivation – correlations must matter • Correlations in multivariate meta-analysis • Impact of errors in approximating within-study covariances

3. Background • Clinical and epidemiological studies typically examine the effect of a treatment or exposure on a number of outcomes • E.g., CVD: effect of a new anti-hypertensive agent on MI, stroke, mortality, etc. • Meta-analyses of these studies will also aim to summarize the evidence about all outcomes • Standard approach: separate meta-analysis for each outcome • Plausible that outcomes are correlated: Common biological pathways, common risk factors, etc. • Alternative: joint meta-analysis of outcomes.

4. Models for Joint Meta-Analyses • For k outcomes: k = 2 Covariance (correlation) Between Effects Across Studies Estimated from the data Covariance between Effectsin each study Assumed known, but covariance rarely reported

5. Motivation • Correlations must matter • Other evidence – eg. MA of longitudinal studies – suggests that it does • But is this a special case? • Consider the general case of two correlated outcomes

6. Objectives • Correlations between treatment effects • Do they reflect dependence in treatment response, or something else? • Approximating within-study covariances • Given that we don’t usually observe estimates of within-study covariances, what is the impact of errors in approximations of within-study covariances on estimates of model parameters?

7. Correlations in MMA Correlations in Multivariate Meta-analysis: Do Correlated Effect Sizes Imply Correlated Effects? Rationale: • Nam et al.: Effect of environmental tobacco smoke (ETS) on incidence of asthma and lower respiratory disease (LRD) • ORAsthma: 1.20 (1.07-1.35), ORLRD: 1.27 (1.10-1.47) • Correlation (between effect sizes): 0.55 (-0.8, 0.99) • How can we interpret this correlation? • Studies that found strong association between ETS and asthma also found strong association with LRD? • Asthma and LRD tend to occur together (regardless of underlying cause)? • People who develop asthma because of ETS are likely to also develop LRD? • Other explanations? Not hard to think of more…

8. Simulation Study • Effect of treatment on two outcomes (A and B) • Generate studies from a process that allows three levels of dependence/correlation: • Events • Treatment response • Studies • Simulate true values - not estimates - of correlation • Since estimates depend on number of studies, sample sizes, etc. • “If correlations were estimated perfectly, would they reflect the true level of dependence in treatment response?”

9. Parameterization of Dependence • Events: P = (P[A], P[B|A], P[B|not A]) • Frequency: Moderate or Common • Dependence: • Independent: P[B|A] = P[B|not A] • Dependent: P[B|A] ≠ P[B|not A] • Strength of dependence: • Moderate or High • Treatment Effects:OR = (OR[A], OR[B|A], OR[B|not A]) • Strength: Moderate or Strong Effect • Dependence in Response: • Independent Effects: OR[B|A] = OR[B|not A] • Dependent Effects: OR[B|A] ≠ OR[B|not A] • Strength of dependence: • Moderate or High

10. Study-Level Dependence • Factors that cause variability between studies will cause correlation within studies • Aspects of design • Study population • Correlated errors Variability in P Correlation between components of P Variability in OR Correlation between components of OR

11. Simulations • Define a scenario • Fix values for P, OR and DP and DOR • Generate large number of studies (N=100K) • Represents pool of studies in which meta-analyses would be carried out • Derive marginal ORs for each study • Calculate correlation between (log) ORs for two events across all studies • Represents true value of correlations estimated in meta-analyses • Does this correlation reflect the strength of the dependence between treatment effects?

12. Observed Correlations Low Correlations btw REs Moderate Correlations btw REs 1 High Correlations btw REs 0.75 0.5 Corr (LOR[A], LOR[B]) 0.25 0 Mod. - Mod. Mod. - Mod. Mod. - Mod. Mod. - Mod. Dep. Dep. Dep. Dep. Mod. - Highly Mod. - Highly Mod. - Highly Mod. - Highly Dep. Dep. Dep. Dep. Common, Mod. Common, Mod. Common, Mod. Common, Mod. Dep. Dep. Dep. Dep. Common, Highly Common, Highly Common, Highly Common, Highly Dep. Dep. Dep. Dep. Moderately Strong, Moderately Moderately Strong, Highly Strong, Moderately Dependent Strong, Highly Dependent  Dependent Dependent

13. Independent Treatment Effects 1 Low Correlations btw REs Moderate Correlations btw REs High Correlations btw REs 0.75 0.5 0.25 0 Mod. - Mod. - Common, Common, Mod. - Mod. - Common, Common, Mod. - Mod. - Common, Common, Mod. Dep. Highly Mod. Dep. Highly Mod. Dep. Highly Mod. Dep. Highly Mod. Dep. Highly Mod. Dep. Highly Dep. Dep. Dep. Dep. Dep. Dep. No Effect on A or B Moderate Effect on A, No Effect on B Strong Effect on A, No Effect on B

14. Summary of Findings • Correlations reflect the general ecological association between outcomes • Within-study correlation was the main driver of the observed correlations • But, even when we assumed independence within studies, correlations under or over-estimated based on commonness and dependence in events • Dependence in treatment response was one of the least influential parameters • Multivariate Normal model for marginal log-ORs not ideal to isolate dependence in treatment response.

15. Approximating Covariances Rationale: • Within-study covariance matrices fixed to observed values to weight observations • Covariance components rarely reported and are approximated or ignored • SABi≈ρ* × SAi× SBi • What if we get ρ*wrong? Impact of Approximating or Ignoring Within-Study Covariances in Multivariate Meta-analyses

16. Simulations • Simulate joint meta-analyses of two outcomes • Generate patient level data so “observed” covariances (SABi) are known • Compare accuracy and precision of results from meta-analyses with known covariances to those using approximations • Overestimate by 50%: ρ* = 1.5ρ • Underestimate by 50%: ρ* = 0.5ρ • Ignore correlations: ρ* = 0*ρ • Wrong direction of association: ρ* = -0.5ρ • Key Simulation Parameter: • Relative size of between-patient and between-study variances and covariances.

17. Results: Si >> D 200 150 100 50 0 Proportional Bias (%) -50 -100 -150 -200 Known Known Known Negative Negative Negative Over-Est Over-Est Over-Est Under-Est Under-Est Under-Est Independent Independent Independent Effect Estimates Btw-Study Variance Correlation

18. Summary • Model is generally robust to poor within-study approximations • Particularly for effect estimates • If only treatment effects are of interest, can do fairly well assuming independence within studies • Estimates of correlations prone to large errors even when covariances known • Bias larger when studies in the meta-analysis were small, D < Si, or D < S • Implications: Can we trust the observed estimates of the correlations?

19. Future Research • Joint analysis of related endpoints in a single study • Same problem with correlations measured with patient-level data? • New application: meta-analysis of composite endpoints • Headache relief within 4 hours without adverse events with migraine therapy • Proposed method: model joint (multinomial) distribution of components • Joint analysis/inference for marginal and composite endpoints

20. Acknowledgements • McGill • Lawrence Joseph • James Hanley • United BioSource • Jaime Caro • Funding: NSERC, CIHR, FRSQ