What evidence informs our practice?

1. What evidence informsour practice? Roger F. Soll, MD H. Wallace Professor of Neonatology Larner College of Medicine, University of Vermont Coordinating Editor, Cochrane Neonatal President, Vermont Oxford Network 1

2. Editorial Team Caitlin O’Connell Assistant Managing Editor Roger F. Soll Coordinating Editor Colleen Ovelman Managing Editor 2

3. Editorial Team Gautham Suresh Baylor University Michael Bracken Yale University Jeffrey Horbar University of Vermont Bill McGuire Hull York Medical School 3

4. Associate Editors 4

5. Guest Discussants Deirdre O'Reilly, MD, MPH Assistant Professor, University of Vermont Director, NPM Fellowship, University of Vermont Danielle Ehret, MD, MPH Assistant Professor, University of Vermont Director, Global Health, Vermont Oxford Network

6. Sponsors Section on Neonatal-Perinatal Medicine

7. What Evidence Informs Our Practice? Roger F. Soll, M.D. is the President of the Vermont Oxford Network and the Coordinating Editor of Cochrane Neonatal No other relevant financial issues to disclose

8. What Evidence Informs Our Practice? To develop an understanding of the strengths and weaknesses of various sources of evidence used to inform our practice of neonatal-perinatal medicine.

9. Evidence Based Medicine “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” Sackett DL, et al. Evidence-Based Medicine: How to Practice and Teach EBM. Churchill-Livingstone. 1997.

10. Evidence Based Medicine

11. Changes in Delivery Room Practice Vermont Oxford Network 2001 to 2009 Soll and colleagues. Obstetric and Neonatal Care Practices for Infants 501 to 1500 g From 2000 to 2009. Pediatrics 2013. 132. 10.1542/peds.2013-0501.

12. Vermont Oxford Network Infants Gestational Age 27 to 29 WeeksInterquartile Ranges 2017 Lowest Quartile 80% 64% 22% 65% 0% Highest Quartile 97% 84% 56% 97% 36% Antenatal Steroids Cesarean Delivery DR CPAP Tracheal Intubation DR Surfactant Over 22,000 Infants at NICUs in the Vermont Oxford Network

13. Vermont Oxford Network Infants Gestational Age 27 to 29 WeeksInterquartile Ranges 2017 Lowest Quartile 0% 7% 0% 0% Highest Quartile 11% 31% 6% 7% Mortality CLD @ 36 wks Pneumothorax Severe IVH Over 22,000 Infants at NICUs in the Vermont Oxford Network

14. Evidence Based Medicine If we are all reading the same information… Why aren’t we operating from the same playbook?

15. What evidence should I use to inform my practice?

16. The Spectrum of Evidence Strength of the evidence: Perry Mason • Multiple eyewitnesses • One sober eyewitness who got a good look • Physical evidence at the crime scene • Pattern of previous criminal activity • Round up the usual suspects

17. The Evidence Hierarchy EBM hierarchies rank study types based on the strength and precision of their research methods. Different hierarchies exist for different question types and experts may disagree on the exact rank of information in the evidence hierarchies. However, most agree that current, well designed systematic reviews and meta-analyses are at the top of the pyramid, and that expert opinion and anecdotal experience are at the bottom.

18. Classifying the Quality of the Evidence 1. Systematic review of multiple well designed randomized controlled trials. 2. Properly designed randomized controlled trial of appropriate size. 3. Well-designed trials without randomization 4. Well-designed non-experimental studies 5. Opinions of respected authorities (based on clinical evidence, descriptive studies or reports of expert committees) (no longer seen as “evidence”)

19. The Evidence Hierarchy

20. Hierarchy of Study Designs for Intended Effects of Therapy • Randomized Controlled Trials • Prospective Follow-up Studies • Retrospective Follow-up Studies • Case-Control Studies • Anecdotal: Case Reports and Series • Hierarchy of Study Designs for Discovery and Explanation • Anecdotal: Case Reports and Series • Case-Control Studies • Retrospective Follow-up Studies • Prospective Follow-up Studies • Randomized Controlled Trials Vandenbroucke JP (2008). "Observational research, randomised trials, and two views of medical science". PLoS Med. 5 (3): e67. doi:10.1371/journal.pmed.0050067. PMC 2265762. PMID 18336067.

21. The strengths and weaknesses of the evidence clinical impressions case series without formal controls studies with formal controls randomized controlled trials meta-analyses Evidence Based Medicine

22. Fundamentally the source of all discovery Also, the source of all erroneous conclusions Clinical Impressions

23. One case: “in my experience…” Two cases: “in case after case…” “time and again…” Three cases: “in my series…” Clinical Impressions

24. During the 1960’s, obstetricians interested in parturition noted that lambs whose mothers were given corticosteroids in an attempt to initiate labor, had less respiratory distress and increased survival… followed by detailed randomized controlled trials both in the animal model and in humans. Clinical Impressions

25. During the 1950’s, obstetricians reported the successful use of diethylstilbestrol (DES) in maintaining the pregnancy of a woman who previously had a series of miscarriages…. followed by widespread use despite limited data from randomized controlled trials Clinical Impressions

26. Sometimes the outcome of current forms of therapy can be predicted with such certainty that past experience provides a valid basis for interpreting current observation Cases and Case SeriesWithout Formal Controls

27. Hazardous Journeys Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials Smith Gordon C S, Pell Jill P. BMJ 2003; 327 doi: https://doi.org/10.1136/bmj.327.7429.1459

28. Hazardous Journeys Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials Objectives: To determine whether parachutes are effective in preventing major trauma related to gravitational challenge. Design: Systematic review of randomized controlled trials. Data sources: Medline, Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists. Study selection: Studies showing the effects of using a parachute during free fall. Main outcome measure: Death or major trauma, defined as an injury severity score > 15. Results: We were unable to identify any randomized controlled trials of parachute intervention.

29. Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomized controlled trials. Advocates of evidence based medicine have criticized the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organized and participated in a double-blind, randomized, placebo controlled, crossover trial of the parachute.

30. Look Before You Leap Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial PArticipation in RAndomized trials Compromised by widely Held beliefs aboUt lack of Treatment Equipoise (PARACHUTE) Trial Yeh and colleagues. BMJ 2018; 363: k5094

31. Look Before You Leap Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial Yeh and colleagues. BMJ 2018; 363: k5094

32. Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial Yeh and colleagues. BMJ 2018; 363: k5094 Parachute use did not significantly reduce death or major injury (0% for parachute v 0% for control; P > 0.9). This finding was consistent across multiple subgroups. Compared with individuals screened but not enrolled, participants included in the study were on aircraft at significantly lower altitude (mean of 0.6 m for participants v mean of 9146 m for non-participants; P < 0.001) and lower velocity (mean of 0 km/h v mean of 800 km/h; P < 0.001). “Representative study participant jumping from aircraft with an empty backpack. This individual did not incur death or major injury upon impact with the ground”

33. If a particular form of care is associated with an abnormal outcome that has not been seen before, observations without formal controls may be an adequate basis for abandoning the form of care in question Cases and Case SeriesWithout Formal Controls

34. The Thalidomide Disaster Thalidomide was first marketed in 1957 in West Germany. Primarily prescribed as a sedative or hypnotic, thalidomide also claimed to cure "anxiety, insomnia, gastritis, and tension". Afterwards, it was used against nausea and to alleviate morning sickness in pregnant women. Thalidomide became an over-the-counter drug in West Germany on October 1, 1957.

35. Shortly after the drug was sold in West Germany, between 5,000 and 7,000 infants were born with phocomelia (malformation of the limbs). Only 40% of these children survived.

36. “Heroine” of FDA Keeps Bad Drug Off of Market Washington Post 1962 Frances Oldham Kelsey, PhD

37. E-Ferol Injection In April 1984, the US FDA was notified of an unusual clinical syndrome consisting of ascites, liver and renal failure, thrombocytopenia, and death among low birth weight infants exposed to an intravenous vitamin E preparation, E-Ferol. The product, which had not been tested for safety prior to marketing, was voluntarily withdrawn from the market in early April.

38. Morbidity and mortality among low birth weight infants exposed to an intravenous vitamin E product, E-Ferol. To further investigate the reported associations, the FDA conducted a retrospective cohort study among seven neonatal intensive care units where the product had been used. Included in the study were 379 infants weighing 2,000 grams or less and surviving at least two days; 148 (39%) had been exposed to E-Ferol. Compared with the unexposed infants, the exposed infants were more likely to die and to have ascites, hepatomegaly, thrombocytopenia, and a combination of clinical events similar to the syndrome initially reported. Arrowsmith and colleagues. Pediatrics. 1989 Feb;83(2):244-9.

39. although past experience suggests that caution is appropriate in interpreting the results of uncontrolled case series, it is often not exercised prone to selective reporting enthusiasm for treatment has been shown to be inversely related to number of patients studied Cases and Case SeriesWithout Formal Controls

40. “Therapeutic reports with controls tend to have no enthusiasm, and reports with enthusiasm tend to have no controls” - Sackett 1986 Evidence Based Medicine

41. Historical controls Case controls Non-randomized concurrent controls Randomized controls Studies With Formal Controls

42. Comparison between people who have received a relatively recently introduced form of care with other individuals cared for in a different way during an earlier era Historical Controls

43. Red blood cell transfusion, feeding and necrotizing enterocolitis in preterm infants. OBJECTIVE: Preliminary studies suggested an association between red blood cell (RBC) transfusion and necrotizing enterocolitis (NEC) in premature neonates. An advantageous effect of withholding feeds during transfusion has never been studied. We aimed, first, to determine whether preterm infants who developed NEC were more likely to be transfused in the 48 to 72 h before the diagnosis of NEC; second, to test if a strict policy of withholding feeds during transfusion would decrease the incidence of transfusion-associated NEC. El-Dib M and colleagues. Red blood cell transfusion, feeding and necrotizing enterocolitis in preterm infants. J Perinatol. 2011 Mar;31(3):183-7. doi: 10.1038/jp.2010.157.

44. Red blood cell transfusion, feeding and necrotizing enterocolitis in preterm infants. STUDY DESIGN: The study was conducted in two phases. Phase 1: a retrospective case-control study of premature low birth weight (< 32 weeks and < 2500 grams) infants who developed NEC over a 6-year period. Phase 2: a comparison study of the incidence of NEC during the 18-months preceding, and the 18 months following the change of practice to withholding feeds during RBC transfusion. El-Dib M and colleagues. Red blood cell transfusion, feeding and necrotizing enterocolitis in preterm infants. J Perinatol. 2011 Mar;31(3):183-7. doi: 10.1038/jp.2010.157.

45. The incidence of necrotizing enterocolitis during the 18-months preceding and the 18 months following the change of practice to withholding feeds during RBC transfusion. Implementing the policy of withholding feeds during transfusion was associated with a decrease in the incidence of NEC from 5.3 to 1.3% (P=0.047). El-Dib M and colleagues. Red blood cell transfusion, feeding and necrotizing enterocolitis in preterm infants. J Perinatol. 2011 Mar;31(3):183-7. doi: 10.1038/jp.2010.157.

46. Randomized (RCT) vs Historical (HCT) Controlled Trials Six Therapies: 50 RCT’s, 56 HCT’s 79% Historical Controlled Trials supported intervention 20% Randomized Controlled Trials supported intervention Sacks 1982 Historical Controls

47. When substantial differences in outcome are noted between two different time frames, this may only reflect changes in other undocumented factors that have modified outcome Inferences based solely on studies that use historical controls tend to lead to conclusions that new forms of care are effective, when less biased comparisons suggest that they are not, or that the estimate is exaggerated Historical Controls

48. Studies that start by identifying persons with and without a disease of interest (cases and controls, respectively) and then look back in time to find differences in exposure to risk factors. Case Controls

49. Of 187 newborns admitted to a 33-bed, level III neonatal intensive care unit between 1/1/1985 and 6/23/1985, 33 developed necrotizing enterocolitis during their hospital stay. A case-control study, with matching on birth weight, date of admission and duration of stay, failed to reveal any association of the syndrome with type or timing of feeding, perinatal hypoxic events, or specific microbial organisms. By contrast, however, transfusion of packed red blood cells was highly and significantly associated with the syndrome (odds ratio = 15.1, 95% confidence interval = 2.59 to 92.51). During this outbreak period, the incidence of necrotizing enterocolitis by birth weight was 30.6% in infants less than 1,500 gm, 10.8% in infants 1,500 to 2,500 grams, and 11.9% in infants 2,500 gm or more. An outbreak of necrotizing enterocolitis. Association with transfusions of packed red blood cells. McGrady GA and colleagues. Am J Epidemiol. 1987 Dec;126(6):1165-72.

50. Comparison of two or more cohorts of individuals who happen to have received alternative forms of care concurrently Non-Randomized Concurrent Controls