Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine

1. Understanding systematicreviews and meta-analyses inneonatal-perinatal medicine Roger F. Soll, MD H. Wallace Professor of Neonatology Larner College of Medicine, University of Vermont Coordinating Editor, Cochrane Neonatal President, Vermont Oxford Network 1

2. Editorial Team Caitlin O’Connell Assistant Managing Editor Roger F. Soll Coordinating Editor Colleen Ovelman Managing Editor 2

3. Editorial Team Bill McGuire Hull York Medical School Co-coordinating Editor Cochrane Neonatal 3

4. Editorial Team New and Improved! The “Co-Co” 3

5. Editorial Team Gautham Suresh Baylor University Michael Bracken Yale University Jeffrey Horbar University of Vermont Prakeshkumar Shah University of Toronto 3

6. Associate Editors 4

7. Guest Discussants Deirdre O'Reilly, MD, MPH Assistant Professor, University of Vermont Director, NPM Fellowship, University of Vermont Danielle Ehret, MD, MPH Assistant Professor, University of Vermont Director, Global Health, Vermont Oxford Network

8. Sponsors Section on Neonatal-Perinatal Medicine

9. Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine Roger F. Soll, M.D. is the President of the Vermont Oxford Network and the Coordinating Editor of Cochrane Neonatal No other relevant financial issues to disclose

10. Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine To develop an understanding of the strengths and weaknesses of evidence provided by systematic reviews and meta-analyses to inform our practice of neonatal-perinatal medicine.

11. The Evidence Hierarchy What evidence should I use to inform my practice?

12. Understanding systematic reviews and meta-analysis Akobeng AK Archives of Disease in Childhood 2005;90:845-848.

13. “Narrative Reviews” vs. “Systematic Reviews” Review articles in the medical literature have traditionally been in the form of “narrative reviews” where experts in a particular field provide what is supposed to be a “summary of evidence” in that field. Narrative reviews, although still very common in the medical field, have been criticized because of the high risk of bias, and “systematic reviews” are preferred. Systematic reviews apply scientific strategies in ways that limit bias to the assembly, a critical appraisal, and synthesis of relevant studies that address a specific clinical question.

14. The problem with traditional “Narrative Reviews” While traditional review articles or narrative reviews can be useful when conducted properly, there is evidence that they are usually of poor quality. Authors of narrative reviews often use informal, subjective methods to collect and interpret studies and tend to be selective in citing reports that reinforce their preconceived ideas or promote their own views on a topic. They are also rarely explicit about how they selected, assessed, and analyzed the primary studies, thereby not allowing readers to assess potential bias in the review process. Narrative reviews are therefore often biased, and the recommendations made may be inappropriate.

15. Search: “Neonate” Limit: Newborn: birth to 1 month; Randomized controlled trial (RCT); Past 10 years 5207 RCT identified

16. Systematic Overview Applies specific research strategies to identify, appraise, and synthesize data from all relevant clinical studies “a study of studies”

17. Meta-analysis Quantitative systematic reviews include meta-analyses: statistical methods to combine the results of similar randomized controlled trials to produce a typical estimate of the effect size

18. Meta-analysis • Meta-analysis demands the same methodological quality expected in a randomized controlled trial: • prospectively designed protocol • comprehensive and explicit search strategy • strict criteria for inclusion of studies • standard definitions of outcomes • standard statistical techniques

19. Meta-analysis • What’s the use of meta-analysis? • increase statistical power • increase precision of estimate • explore differences between study results • create structure for incorporating new evidence

20. Meta-Analysis: Methods Meta-analysis is a two-stage process 1. The first stage involves the calculation of a measure of treatment effect with its 95% confidence intervals (CI) for each individual study. The statistics that are usually used to measure treatment effect include odds ratios (OR), relative risks (RR), and risk differences.

21. Meta-Analysis Methods:Dichotomous Outcomes • Intervention event rate (IER) = rate at which an event occurs in the experimental group • Control event rate (CER) = rate at which an event occurs in the control group • Risk ratio (Relative risk) = the ratio of the risk in the experimental group to the risk in the control group • Risk difference (absolute risk difference) = the difference between the risks in the experimental and control groups • Number needed to treat = number of persons who must be treated for one person to benefit = 1/absolute value of the risk difference

22. Meta-Analysis Methods:Dichotomous Outcomes 95% Confidence Interval (CI) • The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again (and again).

23. Clinically relevant measures of treatment effect • Relative Risk and Relative Risk Reduction measure the strength of the association not the clinical importance • Need to consider frequency of outcome in population • Large RR: evidence of a causal relationship • Large ARD: evidence of actual effect of treatment

24. Meta-Analysis Methods 2. In the second stage of meta-analysis, an overall treatment effect is calculated as a weighted average of the individual summary statistics. In meta-analysis, data from the individual studies are not simply combined as if they were from a single study. Greater weights are given to the results from studies that provide more information, because they are likely to be closer to the “true effect” we are trying to estimate. The weights are often the inverse of the variance (the square of the standard error) of the treatment effect, which relates closely to sample size. The typical graph for displaying the results of a meta-analysis is called a “forest plot”. We will look at “forest plots” later in the discussion today.

25. Appraising a systematic review with or without meta-analysis Although systematic reviews occupy the highest position in the hierarchy of evidence for articles on effectiveness of interventions, it should not be assumed that a study is valid merely because it is stated to be a systematic review. Just as in RCTs, the main issues to consider when appraising a systematic review can be condensed into three important areas: The validity of the trial methodology. The magnitude and precision of the treatment effect. The applicability of the results to your patient or population.

26. Have you ever meta-analysis you didn’t like? Steven Goodman MD, MHS, PhD Annals of Internal Medicine 1991; 114: 244-246

27. Only fair agreement between large clinical trials and meta-analyses LeLorier 1997 Problems with meta-analysis

28. META-ANALYSIS OF MULTIPLE SMALL STUDIES COMPARED TO SINGLE LARGE STUDY ASPIRIN FOR PREVENTION OF PRE-ECLAMPSIA Odds Ratio Decreased Risk Increased CHARACTERISTIC (95% CI) 0.2 0.5 1.0 2.0 4.0 META-ANALYSIS 0.30 (0.20, 0.42) SINGLE LARGE RCT 0.82 (0.72, 1.05) 0.2 0.5 1.0 2.0 4.0 Odds Ratio and 95% CI

29. Problems with meta-analysis • Methodological flaws in meta-analyses • Publication bias • The tendency for investigators to preferentially submit studies with positive results, and the tendency for editors to choose positive studies for publication • Heterogeneity • Concerning variation in the direction or the degrees of results between individual studies

30. Language Bias PROPORTION OF CONTROLLED TRIALS PUBLISHED IN GERMAN AND ENGLISH EGGER 1997

31. Heterogeneity Inevitably, studies brought together in a systematic review will differ. Any kind of variability among studies in a systematic review may be termed heterogeneity. Clinical: Variability in the participants, interventions and outcomes studied Methodological: Variability in study design and risk of bias

32. RCTs “You cannot make a silk purse from a sow’s ear” Systematic Reviews

33. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. BMJ 2009;339:b2535, doi: 10.1136/bmj.b2535

34. Checklist of Items to Include When Reporting a Systematic Review or Meta-Analysis

35. Checklist of Items to Include When Reporting a Systematic Review or Meta-Analysis

36. Cooling for newborns with hypoxic ischaemic encephalopathy Jacobs SE, Berg M, Hunt R, Tarnow-Mordi WO, Inder TE, Davis PG Cochrane Database Syst Rev. 2013 Jan 31;(1):CD003311. doi: 10.1002/14651858.CD003311.pub3. .

37. Cooling for newborns with hypoxic ischaemic encephalopathy Objectives: To determine the effect of therapeutic hypothermia on death and long-term neurodevelopmental disability, and to ascertain clinically important side effects in newborn infants with HIE. Jacobs and colleagues. Cochrane Database Syst Rev. 2013 Jan 31;(1):CD003311. doi: 10.1002/14651858.CD003311.pub3.

38. PICOT Question

39. Cooling for newborns with hypoxic ischaemic encephalopathy PICOT Question In newborn infants with HIE, does therapeutic hypothermia compared to conventional management decrease the risk of death or long-term neurodevelopmental disability at 18 to 24 months of age.

40. Searching databases Bank 14813 items Cast iron bank 805 items Mechanical bank 336 items Book of knowledge bank 14 items

41. Search strategy We use the standard search strategy of Cochrane Neonatal to search Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, OVID Medline; MEDLINE via PubMed for the previous year; and CINAHL. We also search clinical trials databases and the reference lists of retrieved articles for randomized controlled trials and quasi-randomized trials.

42. Searching bibliographic databases 2000 - 2019 Therapeutic hypothermia 14423 items Newborn 1383 items Clinical trials 169 items Randomized controlled trials 127 items

43. Data synthesis   If we identify multiple studies that we consider to be sufficiently similar, we will perform meta-analysis using Review Manager 5 (Review Manager 2014). For categorical outcomes, we will calculate the typical estimates of RR and RD, each with its 95% CI; for continuous outcomes, we will calculate the mean difference or the standardized mean difference, each with its 95% CI. We will use a fixed-effect model to combine data where it is reasonable to assume that studies were estimating the same underlying treatment effect.

44. The forest plot The plot shows, at a glance, information from the individual studies that went into the meta-analysis, and an estimate of the overall results. It also allows a visual assessment of the amount of variation between the results of the studies (heterogeneity).

45. Death or Major Disability in Survivors: Relative Risk

46. Death or Major Disability in Survivors: Risk Difference

47. Annotated Forest Plot Death or Major Disability in Survivors: Relative Risk Confidence interval Point estimate Individual study Typical estimate Line of identity I squared

48. Heterogeneity • Chi2 assesses whether observed differences in results are compatible with chance alone. • I2quantifying inconsistency across studies • A rough guide to interpretation is as follows: •  0% to 40%: might not be important; • 30% to 60%: may represent moderate heterogeneity • 50% to 90%: may represent substantial heterogeneity • 75% to 100%: considerable heterogeneity

49. Page. Research designs in sports physical therapy. International journal of sports physical therapy. Oct 2012;7(5):482‐492.

50. Quality of evidence We use the GRADE approach to assess the quality of evidence for clinically relevant outcomes. We consider evidence from randomized controlled trials as high quality, downgrading the evidence one level for serious (or two levels for very serious) limitations based upon the following: design (risk of bias), consistency across studies, directness of the evidence, precision of estimates, and presence of publication bias.