Department of O UTCOMES R ESEARCH

1. Department of OUTCOMESRESEARCH

2. Clinical Research Design Systematic Reviews and Meta-analysis Daniel I. Sessler, M.D. Michael Cudahy Professor and Chair Department of OUTCOMESRESEARCH The Cleveland Clinic

3. Literature Reviews Reviews are important Often too much primary literature Clinicians cannot critically review all literature Classical reviews Informed synthesis by authors Most helpful when authors are experts and active investigators Excellent perspective Integrates historical development with future directions Typically restricted to best relevant articles Most suitable for reviewing an entire field Subject to author(s) bias

4. Useful for specific interventions & outcomes Specific, important, and sensible question essential Equally effective for complications and therapeutic outcomes Standardized search of all relevant work Documented and reproducible selection process Tabular presentation, often stratified by Research approach Study quality Population Outcome Synthesis can be Qualitative, based on authors’ expertise (and bias) Quantitative: meta-analysis Systematic Reviews

5. Meta-analysis • Statistical summary of systematic review • Effect size and significance • Patient level (patient pooling) or study level (aggregate stats) • Individual patient data preferable, but rarely available • Usually used for randomized trials • Can be used for observational studies— with great caution • Studies must evaluate similar treatment & outcomes • Suitable for various types of data • Dichotomous, continuous, risk difference, relative risk, etc. • Generalizability good; internal validity variable

6. Data-acquisition • Individual studies are unit of analysis • Summary statistics are the data elements • Consider studies to be like patients in a trial • Rigorous a priori inclusion and exclusion criteria • Specify search strategies and sources of studies • Which databases? Search terms? • Language restrictions? • Randomized trials only? • Primary outcomes only? • Published versus unpublished? • Specify adjudication methods

7. Sample Data-extraction Form • Population • Comparison • Treatment • Active vs. placebo • Outcome(s) • Measures of quality • Surprisingly difficult • Adjudication critical

8. Evaluating Study Quality • No good way • Many design errors non-obvious or subtle • Various scoring systems used; points for • Legitimate randomization • Concealed allocation • Blinded outcome evaluation • Drop-outs and reasons described • Standard-of-care: report quality of included studies

9. Reporting Search Results

10. Major Sources of Error • Garbage in, garbage out • Meta-analysis never better than underlying studies • Cannot correct for methodologic errors or bias • Reporting bias • Changed or omitted primary outcomes • Significant findings 2.2-4.7 X more likely to be complete (Dwan 2008) • Subtle (or not) treatment & measurement differences • Publication bias • Large trials are almost always published • Positive studies usually published even if under-powered • Small negative studies less likely than others to be published • Censoring by authors or corporate sponsors • Appropriate editorial decision, but unpublished studies disappear • Meta-analysis depends on knowing about all relevant results

11. Funnel Plots SE of Log(OR) Log(OR)

12. Heterogeneity • Data: variation in study results exceeding chance • Biology: true differences related to methodology • Differences in populations: age, gender, ethnicity, etc. • Differences in drug dose (or drug within a class) • Unappreciated patient factors • Tests: chi square, etc. • Analysis strategies • Minor heterogeneity • Report amount • Combined analysis may be sensible • Treat serious heterogeneity as an interaction • Stratify analysis as for other effect modifiers

13. Analysis Strategies • Fixed-effects model • Assumes all trials share same underlying treatment effect • Treats each trial as random samples from one large trial • Differences in results due to chance alone • Similar to Mantel-Haenszel • Often over-estimates significance • Random-effects model • Assumes each study estimates a unique treatment effect • That is, may truly differ from other included studies • Allows heterogeneity, and is required for heterogeneous data • Weights smaller studies more heavily • Generally provides similar effect estimate with lower precision • More conservative; probably should always be used

14. Forest Plots • Log weighted mean effect ≈ sum of {log (effect)/variance)} for individual studies, divided by sum of 1/variance

15. How Good are Meta-analyses? “Large” defined by n≥1,000 “Large” defined by power Generally, pretty good. But not perfect. Cappelleri, JAMA 1996

16. Meta-analyses Increasingly Common • Most published as part of systematic reviews • Increasingly included in trial reports • Objective comparison of current to previous results • Grant applications • Summarize knowledge • Support equipoise • Need for proposed trial • Complications unlikely Blood loss with low-dose perioperative aspirin

17. Cochrane Collaboration • International non-profit, 1993 • Repository for meta-analyses • Standardized reporting • QUORUM (1999) • PRISMA (2009) • Provides free software • Evidence-based med movement • David Sackett • Gordon Guyatt • Tom Chalmers Archie Cochrane

18. Summary • Systematic reviews • More objective than “expert” reviews • May lack expert perspective and subtlety • Meta-analysis is quantification of systemic review • Subject to major errors • Any problems with underlying studies remain • Publication and reporting bias can be substantial • Heterogeneity can complicate analysis • Conduct and report per guidelines • Useful summary of available literature • Especially when many similar studies are available

19. Department of OUTCOMESRESEARCH