PENN CENTER FOR EVIDENCE-BASED PRACTICE

1. PENN CENTER FOR EVIDENCE-BASED PRACTICE Introduction to Systematic Reviews and Meta-Analyses Craig A Umscheid, MD, MSCE, FACP Assistant Professor of Medicine and Epidemiology Director, Penn Center for Evidence-based Practice Senior Associate Director, ECRI-Penn AHRQ EPC TEACH Level II Workshop 4 NYAM August 8th, 2013

2. Outline • The Case • Definitions • Steps to Consider when Performing or Appraising a SR/MA • Reporting Tool • Conclusions

3. The Case

4. Setting the Stage: Images of the H1N1 Influenza Epidemic

5. 2009 H1N1 Cases Seen in One Mexican Hospital JAMA. 2009;302(17):1880-1887

6. The Case You’re the CMO of a moderate sized hospital in Upstate NY, 100s of miles from a quarternary care hospital, and you hear about the below patient from the Director of your MICU: • 38 yo physician admitted to your hospital with fever, cough, SOB, and muscle aches worsening over three days • PMH of high cholesterol and seasonal allergies, on a statin for cholesterol • SH of 1 drink per day, no tobacco or illicits, 2 kids (3y and 1y), wife, and 2 cats • VS: BP 110/70 HR 115 RR 28 O2 sat 90% • Multilobar pneumonia on CXR, lab tests diagnostic for H1N1 • Admitted to the ICU for hypoxemia • Despite intubation and the use of salvage therapies, hypoxemia persists and patient deteriorates • Within first week of his ICU stay, patient codes and despite heroic efforts by his team he is pronounced dead

7. Demographics of H1N1 Patients in Critical Care JAMA. 2009;302(17):1880-1887

8. Demographics of Hospitalized Pts with H1N1 JAMA. 2009;302(17):1880-1887 JAMA. 2009;302(17):1872-1879

9. Extra Corporeal Membrane Oxygenation (ECMO)

10. The Case (continued) • Should our moderate-sized hospital invest in an ECMO program?

11. Should our moderate-sized hospital invest in an ECMO program? • Yes • No • Don’t know, need more information

12. Definitions

13. “Narrative review” • Traditional review by content expert • Most common method of summarizing a field • Limitations: • Lack systematic methods to identify, appraise, and synthesize information • Potential for selective inclusion and exclusion of studies to support a position

14. “Original research study” • Well designed, well conducted studies can provide unbiased information • Limitations: • False negative results • False positive results • Weak external validity

15. “Systematic reviews” vs. “Meta-analyses” Meta-analysis Statistical pooling of the results of individual studies Aims to produce a single estimate of treatment effect • Systematic review • Literature review prepared using a systematic approach to minimize bias and random error in identification of information • May or may not include a meta-analysis

16. Systematic Review/Meta-analysis • Address limitations of narrative reviews: • Uses prespecified, explicit methods to: • reduce bias • allow for replication • reveal areas of uncertainty or in need of further research • Address limitations of original research studies: • Pools data across studies to: • give more precise estimates of effect • reduce false positive or false negative results • improve generalizability • explore between-study differences in results BMJ. Volume 315. November 22, 1997.

17. Meta-analysis Medline publications

18. Steps to Consider when Performing or Appraising a SR/MA

19. Seven Key Steps for SR/MA • Define the question • Establish eligibility criteria • Search literature • Extract data • Evaluate individual study quality • Synthesize data qualitatively and/or quantitatively (MA) • Explore heterogeneity

20. 1. Define the question • Questions most commonly address therapeutic interventions, harms, and diagnostic tests • Type of question can point to type of studies to seek

21. 1. Define the question (cont) Population • age range, gender, race, diagnosis, severity class Intervention or exposure • specific drug or device, doses, frequencies, IV, PO, inpatient, outpatient, management strategies Comparison group • placebo, usual care, active comparator Outcomes • process measures, clinical outcomes, cost

22. PICO for ECMO SR • Patients • Adults • Acute respiratory failure from H1N1 pneumonia • Any critical care unit in the world • Intervention • ECMO • Comparator • Standard Care • Outcomes • Mortality

23. Seven Key Steps for SR/MA • Define the question • Establish eligibility criteria • Search literature • Extract data • Evaluate individual study quality • Synthesize data qualitatively and/or quantitatively (MA) • Explore heterogeneity

24. 2. Establish eligibility criteria • Study designs • RCT, cohort, case-control, cross-sectional • Publication status • Peer-reviewed publications only • Also include abstracts? • Language • English, other languages? • Other • Publication year • Study size • Minimum follow-up

25. A Note on Publication Bias • Bias can be introduced in a SR by not including relevant studies: • Unpublished (Gray Literature) • Published in non-English language • Published only in meeting proceedings or as abstracts • Published but not identified

26. Cumulative Publication Rate for RCTs Scherer. CDSR. 2007.

27. Submission and Publication Time Ioannidis. JAMA. 1998.

28. Funnel Plots: Assessing Publication Bias No Publication Bias Publication Bias

29. Funnel Plot of Effects of Homeopathy Linde, Lancet, 1997

30. Funnel Plot following Trim and Fill

31. Eligibility criteria for ECMO SR • Only included published studies • Studies were retrieved if: • reported on the use of ECMO in patients with influenza OR • any controlled trial reporting comparisons between those managed with and without ECMO • Studies had to report mortality rates • Minimum of ten patients in each group • No language restrictions

32. Seven Key Steps for SR/MA • Define the question • Establish eligibility criteria • Search literature • Extract data • Evaluate individual study quality • Synthesize data qualitatively and/or quantitatively (MA) • Explore heterogeneity

33. 3. Search literature • Use librarians to help devise search strategy, and filters to limit your search • Identify information resources to search • Electronic literature databases • Reference lists of key papers and review articles • Hand searches of high value journals • Discussion with experts in the field • Contact pharmaceutical or device manufacturers • Trial registries • Use sampling of resources to increase yield • Screen titles, abstracts and full text • Single versus two independent reviewers? • Automatation? • Abstrckr. Available at: http://tuftscaes.org/citation_screening/

34. Search Filters Umscheid. CID. 2013.

35. Electronic Literature Databases • Medline (Medical Literature Analysis and Retrieval System Online) • Premier database of medical research • PubMed interface is free • Ovid interface supported thru institutional accounts, and allows for more complex searching • Embase (ExcerptaMedica Database) • Large European database similar in scope and content to Medline • Includes many conference proceedings • Up to 70% citations in Embase not in Medline • Cochrane Controlled Trials Registry • Fastest, most reliable method for determining if a controlled trial has been published on any topic

36. Search Strategy for ECMO SR

37. Seven Key Steps for SR/MA • Define the question • Establish eligibility criteria • Search literature • Extract data • Evaluate individual study quality • Synthesize data qualitatively and/or quantitatively (MA) • Explore heterogeneity

38. 4. Extract Data • Set priorities for data to extract • Anticipate structure and content of final evidence tables • Resist temptation to extract everything • Establish quality control • Dual extraction vs. solo extraction with random checks • Consider free electronic resources to help • Systematic Review Data Repository. Agency for Healthcare Research and Quality. 2013. Available at: http://srdr.ahrq.gov/.

39. Data extraction for ECMO SR Pulmonary

40. Seven Key Steps for SR/MA • Define the question • Establish eligibility criteria • Search literature • Extract data • Evaluate individual study quality • Synthesize data qualitatively and/or quantitatively (MA) • Explore heterogeneity

41. 5. Evaluate individual study quality • A study’s quality is a measure of its internal validity • Biased studies lead to erroneous findings in systematic reviews • Decide on how to evaluate quality • Use established scales (e.g. Jadad scale for RCTs) • Select components most likely to distinguish studies with higher risks of bias from those with lower risks • Components versus scales • Scales have advantage of providing single summary measure of quality • Multiple scales exist, few are validated, and there are differences in items included and weighting of items across scales • Component analysis has advantage of assessing specific aspects of quality

42. Quality scales Moher et al. Control ClinTrials. 1995.

44. Evaluating quality of ECMO RCTs

45. Study quality (continued) • Decide how quality evaluation will be used • To determine eligibility of study for SR • To weight studies • For subgroup analyses • Subgroup analyses based on quality scales or criteria can help determine effect of bias in studies

46. Examining Effect of LWMH on Risk of DVT Stratified by Study Quality Nurmohamed. Lancet. 1992.

47. Quality of RCTs in ECMO SR

48. Seven Key Steps for SR/MA • Define the question • Establish eligibility criteria • Search literature • Extract data • Evaluate individual study quality • Synthesize data qualitatively and/or quantitatively (MA) • Explore heterogeneity

49. 6. Synthesize data • Qualitative synthesis using evidence tables and written evidence summaries • Quantitative synthesis using meta-analysis • RevMan. Cochrane. 2012. Available at: http://ims.cochrane.org/revman. • OpenMeta[Analyst]. 2012. Available at: http://www.cebm.brown.edu/open_meta.

50. Evidence table for ECMO SR