Systematic Review: Meta-analysis II The nuts and bolts of the statistics

1. Systematic Review:Meta-analysis IIThe nuts and bolts of the statistics Alka M. Kanaya, M.D. Assistant Professor of Medicine, Epi/Biostats April 19, 2007

2. Goals • Understand statistical issues for MA • summary estimate and variance • models • methods • heterogeneity • publication bias • Carry on an intelligent conversation with your statistician • Know if published MA used appropriate methods

3. Meta-analysis: the Steps Formulate a question, eligibility criteria Perform a systematic literature search Abstract the data Perform a statistical analysis Calculate the summary effect size Calculate the summary effect size for subgroups Check for heterogeneity Check for publication bias

4. Clinical Case • 5 y.o. girl c/o ear pain and is found to have an acute otitis media. • Should she get antibiotics? Research Questions: • Are antibiotics effective for pain relief in children with acute OM? • Do antibiotics reduce complications of OM (mastoiditis, hearing problems)?

5. Systematic Review part • Inclusion criteria: • RCT of antibiotic vs. placebo • Children • Without tympanostomy tubes • With OM (regardless of setting of recruitment) • Patient-relevant outcomes • 8 trials with total of 2,287 kids Glasziou, Cochrane Review, 2005

6. Do antibiotics reduce pain? 3 RCTs: Study N RR 1 100 1.41 2 200 0.98 3 300 1.01 Total 700 Goal #1: “Best” estimate • Combine findings from several studies to get the "best" estimate • Calculate weighted mean effect estimate, or a summary effect estimate summary effect estimate= Σ (Ni x effect estimatei) = 640 =0.914 Σ(Ni) 700

7. Goal #1: Calculate weighted mean effect estimate Summary = Σ (weighti x effect estimatei) = 30.3 = 0.99 effect estimate Σ(weighti) 30.5

8. Goal #2: Determine if the summary effect is significant • Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weightsi) Variance of summary estimate = _1_ = .03 30.5 95% CI = + 1.96 √0.03 = + 0.34 Summary OR and 95% CI = 0.99 (0.65 - 1.33)

9. Fixed Effects Random Effects Goal: weighted average of risk from existing studies Goal: estimate the “true” effect Existing studies are the entire population Existing studies are a random sample Weights: variance of individual studies Variance of individual studies + variance of differences between studies Weighti = 1 variance RRi Weighti = 1 + D variance RRi Variance RRs = 1/wi Variance RRs = 1/wi Type of Model?

10. a Summary RR b Summary RR a Summary RR b Summary RR Fixed Effects Model: Random Effects Model:

11. Random VS. Fixed Effects Models Practical Implications of the Choice • Summary estimates: usually similar • Variance: RE model produces large variance of the summary estimate • Confidence intervals: RE model produces wider confidence intervals • Statistical significance: less likely with RE model BOTTOM LINE: • If the individual study findings are similar, the model makes little difference in estimate or statistical significance. • If the individual study findings are heterogeneous, the model can affect the statistical significance.

12. Which method?

13. Which method to use?

14. Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) ai ci Not treated (unexposed) bi di ORi = ai/ci = ai x dilnORmh = Σ (wi x lnORi ) bi/di bix ci Σwi variance lnORi = 1 + 1 + 1+ 1 variance ORmh = 1 ai bi ci di Σ wi weighti = (wi) = 1 variance lnORi 95% CI = elnORmh (1.96 x √variance lnORmh)

15. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1PerforationNo Perforation Antibiotic 1 114 Placebo 3 116 Study 2PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 1. Calculate ORi for each study: OR1= 1 x 116 = 0.34 lnOR1 = -1.08 3 x 114 OR2 = _______ = 0.58 lnOR2 = -0.54

16. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnORi for each study: Varln OR1 = 1 + 1 + 1 + 1 = 1.35 1 3 114 116 Var ln OR2 = ______________ = 0.26 3. Calculate wi for each study: w1 = 1 = 0.74 1.35 w2 = ________ = 3.85

17. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1PerforationNo Perforation Antibiotic 1 114 Placebo 3 116 Study 2PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 4. Calculate the wi x ln ORi for each study: w1 x lnOR1 = 0.74 x -1.08 = -0.80 w2 x lnOR2= 3.85 x -0.54 = -2.08

18. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 5. Calculate the sum of the wi w1 + w2 = 0.74 + 3.85 = 4.59 • Calculate lnORmh =Σ (wi x lnORi) = -0.80 + -2.08 = -0.63Σ wi 4.59 = ORmh = 0.53 • Calculate variance ORmh = 1 = 1 = 0.22 Σ wi 4.59 8. Calculate 95% CI = elnORmh + (1.96 x √ variance lnORmh) = e-.63 + (1.96 x √ 0.22) = 0.21 - 1.34 Summary OR = 0.53 (95% CI 0.21 – 1.34)

19. Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln ORdl = Σ (wix ln ORi) wi = 1 Σwi variancei + D • Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

20. General Variance-Based Method (Fixed or Random Effects) Confidence Intervals: ln ORs = Σ (wi x ln ORi ) wi = 1 ______ Σwi variance lnORi (+D) Variance lnORi = ln ORi / ORl 2 or ln ORu / ORi 2 1.96 1.96 Where ORi = OR on ith study ORl = lower bound of 95 % CI for ith study ORu = upper bound of 95 % CI for ith study • Should always be used for MA of Observational studies  Uses adjusted effect estimates  Preserves adjustment for confounding

21. Randomized Trials Observational Studies • Either Model • Any Method • Either Model • Confidence Interval Method Choice of Model and Method in Meta-Analysis What type of studies are you summarizing?

22. Heterogeneity Are you comparing apples and oranges? Clinical heterogeneity: are studies asking same question? Statistical heterogeneity: is the variation likely to have occurred by chance? Measures how different each individual OR/RR is from the summary OR/RR. Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

23. Problem of Heterogeneity • Study findings are different and should not be combined StudyOR 1 0.01 2 1.0 3 10.0 StudyOR 1 0.35 2 0.56 3 0.97 4 1.15 5 1.75 6 1.95

24. Statistical tests of Homogeneity • Is the variation in the individual study findings likely due to chance? Ho: Effect estimate in each study is the same (or homogeneous) Ha: Effect estimate in each study is not the same (or heterogeneous) Q = Σ(wi x (ln ORmh – ln ORi )2) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous • 8 trials of antibiotics vs. Po for OM, pain outcome: • Q for homogeneity: p=0.91

25. Subgroup & Sensitivity Analysis • Subgroup Analysis – MA of a subgroup of eligible studies  age  ethnicity  risk factors  treatment • Sensitivity Analysis – add or delete questionable studies  eligibility  treatment

26. Subgroup Analysis * p for homogeneity < 0.05

27. Subgroup AnalysisAntibiotics vs. Placebo for acute OMOutcome: abnormal tympanometry

28. Sensitivity Analyses Performed to test the robustness of the findings To fairly assess and acknowledge the limitations Address publication bias (funnel plots, number needed to change result, etc..)

29. Aspirin + Heparin vs. Aspirin alone for Unstable Angina Remove Holdright: RRs = 0.45 (95% CI 0.23 -0.89) ; p-for-hetero=0.71 Add data from two additional trials of LMWH: RRs = 0.56 (95% CI 0.40-0.80); p for heterogeneity: 0.52 Fixed effects model, Mantel-Haenszel method = same findings Sensitivity Analysis

30. Publication Bias • Published studies may not be representative of all studies ever conducted. • Selective publication of studies based on strength & direction of results & language. • AKA “positive outcome bias”

31. Minimizing Publication Bias • Search bibliographies of published papers • Consult with experts • Search for unpublished data • Clinical Trial Registries (NIH, VA) • Institutional Review Boards • Pharmaceutical companies • Hand searches • Consider studies not published in English Stern, BMJ, 2001

32. Statistical Approaches to Publication Bias • Correlation between study sample size (or weight or variance) and effect estimate • Funnel plot • Other fancy statistical methods: • estimate number of unpublished studies that must exist to invalidate the results of the meta-analysis. “File drawer” “Fail-safe N” • eliminate the studies that may have been published due to bias

33. Association of Estrogen use and Endometrial Cancer Correlation of sample size and RR: rho = 0.68; p = 0.08 FUNNEL GRAPH Relative Risk of Endometrial Cancer

34. Correlation of sample size and RR: rho = 0.25; p= 0.64 FUNNEL GRAPH RCTs of Heparin plus ASA vs. ASA Favors Heparin+ASA Favors ASA Sample Size Relative Risk for MI or Death

35. Presentation of the Results Tables: • Study Characteristics  population  sample size  definition of intervention  definition of the outcome  important design features (validity of the data) - randomization - blinding - follow-up - compliance • Study Findings  main and secondary outcomes  outcomes by subgroup  sensitivity analysis findings

36. Table 1. Characteristics of 6 randomized trials of aspirin + heparin vs. aspirin alone to prevent MI and death in patients admitted with unstable angina

37. No Caption Found Antibiotics vs. Placebo in Acute Bronchitis Bent, AMJ, 1999

38. Take Home Messages • You can do a meta-analysis • Good start on becoming an expert in your field • Your work should be reproducible • Your conclusions should be obvious • Include a statistician on you team