Grant Stephen: Chair of the MBC Life Science Informatics Group & CEO, Tessella Inc: grant.stephen@tessella.com

1. Life Science Informatics Group Introduction to Meta-Analysis Christopher H. Schmid, PhD Tufts-New England Medical Center 6 June 2008 Grant Stephen: Chair of the MBC Life Science Informatics Group & CEO, Tessella Inc: grant.stephen@tessella.com Creating Insight & Understanding from Scientific Data

2. The weight of medical knowledge Weight of the Index Medicus According to 10-Year Periods from 1879 to 1977

3. Introduction • Medicine requires hard evidence, often clinical trials • Evidence inconsistent across studies Different study populations Different treatments or protocols Quality of technical design or execution Random variation

4. Reasons for combining data • Get an overall estimate of treatment effect • Appreciate the degree of uncertainty • Appreciate heterogeneity • Forces you to think rigorously about the data

5. Meta-Analysis • A scientific discipline which applies a protocol to critically evaluate and uses statistical methods to combine the results of (previous) research • Provides a quantitative summary of the overall treatment effect (typically an overall estimate and confidence interval) • Increasingly used to understand differences among studies to explain discrepancies of results and to generate hypotheses of interactions • To guide future research

6. Some terms used for meta-analysis • Systematic Review • Overview • Quantitative overview • Research (evidence) synthesis • Research Integration • Pooling (implies lumping data altogether) • Combining (implies performing procedures on data)

7. No one will criticize you for doing a systematic review. But as soon as you combine data, you will most likely have controversy

8. Why is that? • Apples and oranges (heterogeneity) • Garbage in, garbage out (quality) • Selection of outcomes (soft or hard) • Selection of studies • Publication bias • Assumptions used in quantifying results

9. Characteristics of Meta-Analyses • More heterogeneity than multicenter trial • Meta-analysis addresses random variation • Gives pooled estimate of treatment effect • Can be confirmatory or exploratory • Pooling may not be best solution • Need to explain variation

11. Systematic review protocol • Well-focused study question • Identification of studies (design, source, search strategy) • Eligibility criteria (study, patient, and disease characteristics, treatments, outcomes) • Data extraction (definition of outcomes, quality assessment) • Data summary and analysis (outcomes, intention to treat)

12. Issues in formulating a question • Narrow versus broad (for individuals/ subgroups or entire population) • Scientifically meaningful and useful (based on sound biological and epidemiological principles) • Very broadly defined questions may be criticized for mixing apples and oranges • Narrowly focused questions have limited generalizability and may lead to biased conclusions

13. Randomized comparisons of antibiotics for acute sinusitis

14. Literature processing • Questions • Search strategy • Screening of abstracts • Retrieve potential articles • Screen full articles • Data extraction on qualifying articles

15. Issues in Finding and Retrieving Evidence • Search strategies • Sources • Language selection • Published vs. unpublished literature • Use of abstracts • File drawer problem (publication bias) • Multiple publications on same subjects • Disproportionate amount of data for topic

16. Types of Multiple Publications • Overlapping data (preliminary and later reports) • Same data but different authors • Similar data (same authors) but different cohort • need to verify with authors

17. Most meta-analyses are retrospective exercises, suffering from all the problems of being an observational design. We cannot fix bad data.

18. Basic principles in combining data • For each analysis, one study should contribute only one effect • Effect may be single outcome or composite of several outcomes • Effect being combined should be same or similar across studies

19. What kinds of control? • No treatment control • Placebo • Active comparator

20. Types of data that could be combined • dichotomous (events, e.g. deaths) • measures (odds ratios, correlations) • continuous data (mmHg, pain scores) • survival curves • diagnostic test (sensitivity, specificity) • individual patient data • “effect size”

21. Issues in choosing method to combine studies • Metrics • Fixed vs. random effects model • Treatment effect heterogeneity • Baseline rate heterogeneity • Weight

22. Heterogeneity (diversity) • Is it reasonable (are studies and effects sufficiently similar) to estimate an average effect? • Types of heterogeneity • Conceptual (clinical) heterogeneity • Statistical heterogeneity

23. Conceptual (scientific) Heterogeneity Are the studies of similar treatments, populations, settings, design, etc., such that an average effect would be scientifically meaningful?

24. Statistical Heterogeneity Is the observed variability of effects greater than that expected by chance alone?

25. Meta-analyticapproaches • Summary point estimation (random or fixed effect model) • Meta-regression - modeling aggregate data heterogeneity • Baseline risk meta-regression • Response surface - individual patients’ data analysis

26. Sensitivity analyses • Exclude studies • Analyze subgroups • Change assumptions • Use different metric • Compare fixed versus random effects model • Perform cumulative meta-analysis

28. Dichotomous outcomes • Binary outcomes, event or no event, yes or no • Most common type of outcome reported in clinical trials (about 70%) • Examples: dead/alive, stroke/no stroke, cure/failure • 2x2 tables commonly used to report their results • Sometimes continuous variables are forced into dichotomous outcomes. • E.g., a threshold could be used for pain scores and reported as improved or not improved.

29. OR = (a*d) / (b*c)

30. Example of 2x2 table: ISIS-2

31. Available metrics for combining dichotomous outcome data • Odds ratio (OR) • Risk ratio (RR) • Risk difference (RD) • NNT (Number Needed to Treat) can be derived (inverse of the combined risk difference) = 1/RD

32. Calculating treatment effects in ISIS-2 TR = 791/8592 = 0.0921 CR = 1029 / 8595 = 0.1197 RR = 0.0921 / 0.1197 = 0.77 OR = (791 x 7566) / (1029 x 7801) = 0.75 RD = 0.0921 – 0.1197 = -0.028

33. ISIS-2 vascular death estimate & 95% CI Streptokinase vs. Placebo

34. Same change in one scale may have different meaning in another scale

35. Effect of Small Changes (small numbers are fragile)

36. Beta-Blockers after Myocardial Infarction - Secondary Prevention Experiment Control Odds 95% CI N Study Year Obs Tot Obs Tot Ratio Low High === ============ ==== ====== ====== ====== ====== ===== ===== ===== 1 Reynolds 1972 3 38 3 39 1.03 0.19 5.45 2 Wilhelmsson 1974 7 114 14 116 0.48 0.18 1.23 3 Ahlmark 1974 5 69 11 93 0.58 0.19 1.76 4 Multctr. Int 1977 102 1533 127 1520 0.78 0.60 1.03 5 Baber 1980 28 355 27 365 1.07 0.62 1.86 6 Rehnqvist 1980 4 59 6 52 0.56 0.15 2.10 7 Norweg.Multr 1981 98 945 152 939 0.60 0.46 0.79 8 Taylor 1982 60 632 48 471 0.92 0.62 1.38 9 BHAT 1982 138 1916 188 1921 0.72 0.57 0.90 10 Julian 1982 64 873 52 583 0.81 0.55 1.18 11 Hansteen 1982 25 278 37 282 0.65 0.38 1.12 12 Manger Cats 1983 9 291 16 293 0.55 0.24 1.27 13 Rehnqvist 1983 25 154 31 147 0.73 0.40 1.30 14 ASPS 1983 45 263 47 266 0.96 0.61 1.51 15 EIS 1984 57 858 45 883 1.33 0.89 1.98 16 LITRG 1987 86 1195 93 1200 0.92 0.68 1.25 17 Herlitz 1988 169 698 179 697 0.92 0.73 1.18

37. What is the average difference in DBP?

38. (-6.2) + (- 7.7) + (-0.1) -4.7 mmHg = 3 Simple Average ____

39. (554 x -6.2) + (304 x -7.7) + (39 x -0.1) -6.4 mmHg = 554 + 304 + 39 Sample Size Weighted Average

40. General Weighted Average Effect Size where: di = effect size of the ith study wi = weight of the ith study k = number of studies

41. Calculation of weights • Generally the inverse of the variance of treatment effect • Different formula for odds ratio, risk ratio, risk difference • Readily available in books and software

42. Effect Size • Dimensionless metric • Combine standard deviations of diverse types of related effects • Availability and selection of reported effects may be biased • Variable importance of different effects • Frequently used in education, social science literature • Difficult to interpret results

43. Statistical Models of Pooling 2x2 Tables • Fixed Effect Model • Random Effect Model

44. David Bowers. Statistics from scratch. An introduction for Health Care Professionals. John Wiley & Sons, 1996.

45. Container with fixed number of white and black balls(fixed effects model)

46. Random sampling from container with fixed number of white and black balls (different sample size)

48. Summary point estimation Principle - common truth Main Model - fixed effects weighted average Advantages - easy to interpret, applies to whole population Disadvantages - often simplistic; not applicable with heterogeneity

49. Different containers with different proportions of white and black balls(Random effects model)

50. Random sampling from containers to get overall estimate of Ratio of white and black balls