1 / 47

Clinical Epidemiology Boot Camp: Systematic Reviews

Clinical Epidemiology Boot Camp: Systematic Reviews. Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19, 2012. Outline. Introduction – Evidence-Based Medicine (EBM) Levels of evidence To discuss the definition of a systematic review

belden
Download Presentation

Clinical Epidemiology Boot Camp: Systematic Reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Clinical Epidemiology Boot Camp:Systematic Reviews Selina Liu MD MSc FRCPC Research Fellow Resident Research Career Development Program September 19, 2012

  2. Outline • Introduction – Evidence-Based Medicine (EBM) • Levels of evidence • To discuss the definition of a systematic review • vs. traditional/narrative reviews • The process of conducting a systematic review • Strengths & limitations of systematic reviews • To describe how to critically appraise a systematic review • Example of a systematic review

  3. Evidence-Based Medicine • What is Evidence-Based Medicine? • “…the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients” • “ It’s about integrating individual clinical expertise and the best external evidence” • philosophical origins – date back to mid-19th century Paris (or possibly earlier) Sackett DL et al. BMJ. 1996;312(7023):71-2

  4. Evidence-Based Medicine • Five Steps of Evidence-Based Medicine • 1. Asking Focused Questions • Translation of uncertainty to an answerable question • 2. Finding the Evidence • Systematic retrieval of the best evidence available • 3. Critical Appraisal • Testing evidence for validity, clinical relevance, and applicability • 4. Making a Decision • Application of results in practice • 5. Evaluating Performance • Auditing evidence-based decisions Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net

  5. Evidence-Based Medicine • Why Evidence-Based Medicine? • clinical decision making is complex! Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91

  6. Evidence-Based Medicine • How do we practice Evidence-Based Medicine? Can be difficult: • “information overload”  difficult for clinicians to “keep up” with all of the latest evidence • often there are multiple studies examining the same or similar questions • may be of variable quality, generalizability • estimated time required for reading (general medicine): • 19 articles per day, 365 days per year Davidoff F et al. BMJ. 1995;310(6987):1085-6

  7. Evidence-Based Medicine • Weighing the evidence - “Levels of Evidence” OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653

  8. Evidence-Based Medicine of RCTs Systematic reviews of cohort studies

  9. Systematic Reviews • What is a Systematic Review? • the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic • use rigorous, standardized methods for selecting & assessing articles Oxford Centre for Evidence-Based Medicine www.cebm.net/?o=1116 OR • a report that summarizes all evidence that can be drawn from research (or other sources), that is relevant to a specific clinical question

  10. Systematic Reviews • Systematic Reviews vs. Traditional Review Articles • traditional review articles • written by senior expert in the field, summarizes evidence and recommendations • usually address broad areas/questions (i.e. “management of T2DM”) • often lack structure • may include personal experience/anecdotal evidence Fletcher RH & Fletcher SW 2005. Clinical Epidemiology: The Essentials

  11. Systematic Reviews • Systematic Review vs. Traditional/Narrative Review Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380

  12. Systematic Reviews • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  13. Systematic Reviews • Process of Conducting a Systematic Review 1. Define the question 2. Conduct literature search 3. Apply inclusion and exclusion criteria 4. Create data abstraction 5. Conduct analysis • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  14. Systematic Reviews - Process • 1. Define the Question • single, focused question • i.e. What is the effect of cinnamon on glycemic control in diabetes? • specify inclusion and exclusion criteria • Population, Intervention or Exposure, Outcome, Methodology • For the systematic review to be useful: • strong studies of the question should be available, but their results should not be so much in agreement that the question is already answered! • there should not be so few studies of the question that each individual study could be fully critiqued directly

  15. Systematic Reviews - Process • 2. Conduct literature search • need to ensure that all of the appropriate studies are included • NOT just a biased sample of studies • decide on information sources • i.e.MEDLINE, recent reviews, textbooks, experts in the field, articles cited by references already found by other approaches, databases of articles, clinical trial registries etc. • identify titles and abstracts

  16. Systematic Reviews - Process • 3. Apply inclusion and exclusion criteria • Apply inclusion and exclusion criteria to titles and abstracts • obtain full articles for eligible titles and abstracts • Apply inclusion and exclusion criteria to full articles • Select final eligible articles • Assess agreement on study selection • of the initial titles and abstracts retrieved, usually only a small proportion of articles are selected

  17. Systematic Reviews - Process • 4. Create data abstraction • Assess methodologic quality of each article • Assess agreement on validity assessment • Data abstraction • Participants • Interventions and Comparison Interventions • Study Design • Results

  18. Systematic Reviews - Process • 5. Conduct analysis • Summarize data • If appropriate: meta-analysis – statistical technique to combine quantitative data • usually combine studies vs. combine patients • Describe results – often graphically • Forest Plot – shows point estimate and confidence interval (for RCTs, observational studies) • Summary Receiver-Operator Curves (for studies of diagnostic tests) • Explore heterogeneity, conduct subgroup analysis (if appropriate) • Explore possibility of publication bias (and other biases)

  19. Systematic Reviews - Process • How to decide if appropriate to perform a meta-analysis? • Two general approaches: • 1. statistical test for homogeneity • BUT – even if fail to reject H0 (i.e. no evidence of a statistically significant difference between studies), usually have high risk of false-negative (saying studies are homogeneous when they really are not) • Limited power - meta-analyses are usually of few number of studies, - affected also by number of patients/study, distribution of patients among studies • 2. informed judgement

  20. Systematic Reviews - Process • Meta-analysis – mathematical models: • Fixed-Effect Model • Assumes that studies are of exactly the same question, so results differ only by chance • Confidence intervals calculated may imply more precision (i.e. are narrower) than in reality (since studies usually differ somewhat) • Random-Effects Model • Assumes that the studies address somewhat different questions, but that they form a closely related family of studies of a similar question • Studies taken to be a random sample of all studies bearing on the question • Produces WIDER confidence intervals (more “realistic”) Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4th Edition)

  21. Systematic Reviews – Forest Plot

  22. Systematic Reviews - Bias • Several types of bias: • publication bias • published studies may be systematically different than unpublished studies (“positive” studies vs. “negative” studies?) • language bias • i.e. if only English-language articles are selected • size bias • large studies that result in several publications may be more readily noticed than smaller studies • bias related to funding? • industry-sponsored studies

  23. How to detect publication bias? • Funnel plots – plot effect vs. study size/precision symmetrical, peaked distribution (inverted funnel) • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  24. How to detect publication bias? • Funnel plots asymmetrical distribution • Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

  25. Systematic Reviews - Strengths • provide an efficient way to become familiar with the best available research evidence for a focused clinical question • can establish whether results are consistent, generalizable across populations/settings, treatment variations, and whether findings vary by certain subgroups • can extend the available literature (if the review team has obtained unpublished information from the primary authors) • meta-analyses – may provide a more precise estimate of the underlying “true effect” than any individual study Garg AX, Hackam D, Tonelli M. 2008. Clin J Am SocNephrol. 3(1):253-60.

  26. Systematic Reviews - Limitations • summarized results are limited by the quality of the primary studies • “garbage in garbage out” • results dependent on selection of included articles • quality threshold, publication bias, language bias etc. • meta-analyses - may be inappropriate to mathematically combine primary study results if the primary studies differ in design, quality, population, intervention etc. • subjectivity involved in deciding whether to pool or not • subjectivity in interpretation of summarized results (“over-interpretation) Garg AX, Hackam D, Tonelli M. 2008. Clin J Am SocNephrol. 3(1):253-60.

  27. Systematic Reviews – Critical Appraisal Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71 • Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33

  28. Critical Appraisal – Tools Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157

  29. Critical Appraisal – Tools Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157

  30. Critical Appraisal – Tools • AMSTAR – 2007 • Assessment of Multiple SysTemAticReviews • Shea BJ, Grimshaw JM, Wells GA et al. • 11 item tool • developed via exploratory factor analysis of a 37-item assessment tool • Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

  31. Critical Appraisal - Tools • Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

  32. Reporting Systematic Reviews - Tools • The PRISMA Statement – 2009 • Preferred Reporting Items for Systematic reviews and Meta-Analyses • Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group • PLoS Medicine • Annals of Internal Medicine • BMJ • Journal of Clinical Epidemiology • Open Medicine • International Journal of Surgery

  33. The PRISMA Statement - 2009 • Aim – to help authors improve the reporting of systematic reviews and meta-analyses • **NOT intended to be a quality assessment tool • 27 item checklist • four-phase flow diagram • update and expansion of prior QUOROM statement • QUality Of Reporting Of Meta-analyses - 1996 • focused on reporting of meta-analyses of randomized controlled trials

  34. Table 1. Checklist of items to include when reporting a systematic review or meta-analysis. Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

  35. Figure 1. Flow of information through the different phases of a systematic review. Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

  36. Example of a Systematic Review Cochrane Database of Systematic Reviews2012, Issue 9:CD007170

  37. Cinnamon for Diabetes Mellitus • Objective – to evaluate the effects of cinnamon in patients with diabetes mellitus • Selection criteria – all RCTs comparing the effects of orally administered monopreparations of cinnamon (Cinnamomum spp.) to placebo, active medication or no treatment in persons with either type 1 or type 2 diabetes mellitus • Primary outcomes – fasting blood glucose, post-prandial glucose, adverse events • Secondary outcomes – HbA1c, serum insulin, HOMA-IR, HRQoL, morbidity, costs Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  38. Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  39. Risk of Bias Summary • Review authors’ judgements about each “risk of bias” item for each included study Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  40. Risk of Bias Graph • Review authors’ judgements about each “risk of bias” item presented as percentages across all included studies Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  41. Results – Cinnamon vs. Placebo • Primary outcome – fasting blood glucose Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  42. Results – Cinnamon vs. Placebo • Primary outcome – adverse events Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  43. Results – Cinnamon vs. Placebo • Secondary outcome – HbA1c Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  44. Results • Other primary outcomes: • Post-prandial glucose – 1trial • Other secondary outcomes: • Serum insulin – 2 trials • HOMA-IR – 2 trials • HRQoL, morbidity, costs – no trials Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  45. Conclusions • Insufficient evidence to support the use of cinnamon for type 1 or type 2 diabetes • Further trials required: • To address methodologic issues in current studies • i.e. allocation concealment, blinding • To include other important endpoints • HRQOL, diabetes complications, cost Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170

  46. Useful Resources • Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill • available online via Western Libraries • Oxford Centre for Evidence-Based Medicine (CEBM) • www.cebm.net • Cochrane Database of Systematic Reviews • www.thecochranelibrary.com

  47. References • Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS. BMJ. 1996;312(7023):71-2 • Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91 • Davidoff F, Haynes B, Sackett D, Smith R. BMJ. 1995;310(6987):1085-6 • Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net • OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653 • Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4thEdition). Baltimore MD, Lippincott Williams & Wilkins • Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill • Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380 • Garg AX, Hackam D, Tonelli M. 2008. Clin J Am SocNephrol. 3(1):253-60. • Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71 • Tonelli M, Hackam D, Garg AX. Methods Mol Biol.2009;473:217-33 • Cochrane Database of Systematic Reviews www.thecochranelibrary.com

More Related