1 / 51

Grading quality of evidence the GRADE approach

Society of General International Medicine 32 nd Annual Meeting, May 14 th 2009 Elie A. Akl, MD, MPH, PhD David Atkins, MD, MPH Eric Bass, MD, MPH Yngve Falck-Ytter, MD Stephanie Chang, MD, MPH . Grading quality of evidence the GRADE approach . Session outline.

cortez
Download Presentation

Grading quality of evidence the GRADE approach

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Society of General International Medicine 32nd Annual Meeting, May 14th 2009 Elie A. Akl, MD, MPH, PhD David Atkins, MD, MPH Eric Bass, MD, MPH Yngve Falck-Ytter, MD Stephanie Chang, MD, MPH Grading quality of evidencethe GRADE approach

  2. Session outline • Introductions, objectives (5 min) • Overview of the GRADE approach (25 min) • Applying the GRADE approach (45 min) • Wrap-up (10 min) • Session evaluation (5 min)

  3. Disclosure • Presenters are members of the GRADE working group and have received honoraria related to this work that were deposited into research accounts • No conflict of interest related to pharmaceutical industry

  4. Objectives

  5. Learning objectives • To enumerate GRADE categories for quality of evidence • To list the GRADE factors that affect the quality of evidence • To apply the GRADE approach to a specific body of evidence • To discuss the strengths and limitations of the GRADE approach

  6. Overview of the GRADE approach

  7. Grades of Recommendation Assessment, Development and Evaluation GRADE Working Group CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008

  8. GRADE definition of Quality of Evidence “Extent to which confidence in estimate of effect adequate to support decision”

  9. GRADE rating of outcomes • GRADE rates the quality of evidence for each outcome separately • The type of evidence may be different for different outcomes • Different audiences are likely to have varying perspective on the importance of outcomes • GRADE considers desirable and undesirable outcomes and rates their relative importance

  10. GRADE rating of outcomes • Desirable outcomes • lower mortality • reduced hospital stay • reduced duration of disease • reduced resource expenditure • Undesirable outcomes • adverse reactions • the development of resistance • costs of treatment

  11. Critical for decision making Important, butnot critical for decision making Of low importance GRADE rating of outcomes 9 8 7 6 5 4 3 2 1

  12. GRADE rating of outcomes • Ranking outcomes by their relative importance can help to focus attention on those outcomes that are considered most important • Outcome choice should be based on what is important, and not what was measured

  13. GRADE uses a comprehensive and transparent conceptual framework for rating the quality of evidence

  14. GRADE levels of Evidence • High: • Moderate: • Low: • Very low:

  15. GRADE levels of Evidence • High: considerable confidence in estimate of effect • Moderate: further research likely to have impact on confidence in estimate, may change estimate • Low: further research is very likely to impact on confidence, likely to change the estimate • Very low: any estimate of effect is very uncertain

  16. Determinants of quality • Quality starts high for evidence from RCTs • Quality starts low for evidence from observational studies • 5 factors lower the quality of evidence • 3 factors can increase the quality of evidence

  17. Factors that lower quality • Study limitations (in design and execution) • Inconsistency • Indirectness • Reporting bias • Imprecision

  18. Factors that lower quality • Study limitations (in design and execution) • Inappropriate randomization • Lack of concealment • Intention to treat principle violated • Inadequate blinding • Loss to follow-up • Early stopping for benefit

  19. Factors that lower quality • From Cates , CDSR 2008 CDSR 2008

  20. Factors that lower quality Overall judgment required

  21. Factors that lower quality • Inconsistency • Assess for inconsistency (Heterogeneity) • variation in size of effect • overlap in confidence intervals • statistical significance of heterogeneity • I2 • If inconsistency  look for explanation • patients, intervention, outcome, methods • If unexplained inconsistency  downgrade quality

  22. Factors that lower quality • Inconsistency • Heparin or vitamin K antagonists for survival in patients with cancer: Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.

  23. Factors that lower quality • Inconsistency • Non-steroidal drug use and risk of pancreatic cancer: Capurso G, Schünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, DelleFave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories. Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.

  24. Factors that lower quality • Indirectness of Evidence • Differences in populations/patients • mild versus severe COPD • Differences in interventions • all inhaled steroids, new vs. old • Differences in outcomes • important vs. surrogate;

  25. Factors that lower quality • Indirectness of Evidence • indirect comparisons • interested in A versus B • have A versus C and B versus C Alendronate Risedronate Placebo

  26. Factors that lower quality • Publication bias • Number of small studies • Faster and multiple publication of “positive” trials • Fewer and slower publication of “negative” trials

  27. ISIS-4Lancet 1995 I.V. Mg in acute myocardial infarction Meta-analysisYusuf S.Circulation 1993 Publication bias Egger M, Smith DS. BMJ 1995;310:752-54

  28. Funnel plot 0 Symmetrical: No publication bias 1 Standard Error 2 3 0.1 0.3 0.6 1 3 10 Odds ratio Egger M, Cochrane Colloquium Lyon 2001

  29. Funnel plot 0 Asymmetrical: Publication bias? 1 Standard Error 2 3 0.1 0.3 0.6 1 3 10 Odds ratio Egger M, Cochrane Colloquium Lyon 2001

  30. ISIS-4Lancet 1995 I.V. Mg in acute myocardial infarction Meta-analysisYusuf S.Circulation 1993 Publication bias Egger M, Smith DS. BMJ 1995;310:752-54

  31. Meta-analysis confirmed by mega-trials Egger M, Smith DS. BMJ 1995;310:752-54

  32. Factors that lower quality • Imprecision • small sample size • small number of events • wide confidence intervals • uncertainty about magnitude of effect • how to decide if CI too wide? • grade down one level? • grade down two levels?

  33. Factors that raise quality • Large magnitude of effect • Dose response relation • All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

  34. Factors that raisequality • Large magnitude of effect • large (RRR 50%) can raise by one level • very large (RRR 80%) can raise by two levels • common criteria • everyone used to do badly • almost everyone does well • Examples • oral anticoagulation for mechanical heart valves • insulin for diabetic ketoacidosis • hip replacement for severe osteoarthritis

  35. Factors that raisequality • Dose response relation • higher INR – increased bleeding • childhood lymphoblastic leukemia • risk for CNS malignancies 15 years after cranial irradiation • no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%)

  36. Factors that raise quality 3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

  37. Factors that raise quality • Example 1: higher death rates in private for-profit versus private not-for-profit hospitals • patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals • for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect)

  38. Factors that raise quality • Example 2: hypoglycaemic drug phenformin causes lactic acidosis • The related agent metformin is under suspicion for the same toxicity. • Large observational studies have failed to demonstrate an association • Clinicians would be more alert to lactic acidosis in the presence of the agent

  39. Summary of GRADE framework for rating the quality of evidence

  40. Evidence Profiles and Summary of Findings (SoF) Tables summarize the rating of the quality of evidence across selected outcomes

  41. Applying the GRADE approachExercise: parenteral anticoagulation for prolonging the survival of patients with cancer

  42. Wrap-up

  43. Advantages of GRADE • Developed by a widely representative group of international guideline developers • Clear separation between quality of evidence and strength of recommendations • Explicit evaluation of the importance of outcomes • Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings

  44. Advantages of GRADE • Transparent process of moving from evidence to recommendations • Explicit acknowledgment of values and preferences • Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients, and policy makers • Useful for systematic reviews and health technology assessments, as well as guidelines

  45. Disadvantages of GRADE • Involves a number of judgments that might affect its reliability • Requires expertise/training

  46. Session evaluation

More Related