1 / 50

Grades of Recommendation Assessment, Development and Evaluation

Introduction to GRADE Methodology with emphasis on guideline development Cochrane Review Group on HIV/AIDS University of California, San Francisco. Grades of Recommendation Assessment, Development and Evaluation.

cain
Download Presentation

Grades of Recommendation Assessment, Development and Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to GRADE Methodologywith emphasis on guideline developmentCochrane Review Group on HIV/AIDSUniversity of California, San Francisco

  2. Grades of Recommendation Assessment, Development and Evaluation CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008, Lancet ID 2007, PLOS Medicine 2007

  3. GRADE Uptake • World Health Organization • Allergic Rhinitis in Asthma Guidelines (ARIA) • American Thoracic Society • British Medical Journal • Infectious Disease Society of America • American College of Chest Physicians • UpToDate • American College of Physicians • Cochrane Collaboration • National Institute Clinical Excellence (NICE) • Infectious Disease Society of America • European Society of Thoracic Surgeons • Clinical Evidence • Agency for Health Care Research and Quality (AHRQ) • Others

  4. Guideline development process

  5. Guideline development process GRADE

  6. PICO Questions • Population, Intervention, Comparator, Outcome • Sometimes given as PICOT (T= Time-frame) • Precise PICO questions = central to GRADE • Saves resources in developing new guidelines: • Helps limiting scope of project • Specifies the search strategy more clearly • Guides data extraction • Helps with formulating recommendations • A priori PICO questions limit the risk of selection bias

  7. Example of PICO Question (example from 2010 MSM Guidelines process)

  8. Quality of evidence • The quality of evidence reflects the extent to which confidence in an estimate of the effect is adequate to support recommendations. 9

  9. Quality of evidence by outcome • GRADE rates the quality of evidence for each outcome separately • The source of evidence may be different for different outcomes • The same source of evidence can provide varying quality of evidence for the different outcomes 10

  10. GRADE levels of Evidence • High: • Moderate: • Low: • Very low:

  11. GRADE levels of Evidence • High: considerable confidence in estimate of effect • Moderate: further research likely to have impact on confidence in estimate, may change estimate • Low: further research is very likely to impact on confidence, likely to change the estimate • Very low: any estimate of effect is very uncertain

  12. Outcome #1 Quality: High Outcome #2 Quality: Moderate Outcome #3 Quality: Low I B II V III GRADE Quality of evidence across studies Old systems

  13. Components determining quality • RCTs start high • Observational studies start low What lowers quality of evidence? 5 factors: Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias

  14. Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias Assessment of detailed design and execution (risk of bias) For RCTs: • Lack of allocation concealment • No true intention to treat principle • Inadequate blinding • Loss to follow-up • Early stopping for benefit

  15. Cochrane risk of bias graph, assessing overall risk of bias across several studies 16

  16. Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias • Judgment • variation in size of effect • overlap in confidence intervals • statistical significance of heterogeneity • I2 (or 2) • Look for explanation for inconsistency • patients, intervention, comparator, outcome, methods

  17. Heterogeneity Any intervention vs control Ng BE, Butler LM, Horvath T, Rutherford GW. Population-based biomedical sexually transmitted infection control interventions for reducing HIV infection. Cochrane Database Syst Rev. 2011 Mar 16;(3):CD001220. 18

  18. Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias • Indirect comparisons • Interested in head-to-head comparison • Drug A versus drug B – but what if not studied? • Study populations • Differences in • patients • interventions • comparator (e.g., differences in dose) • outcomes

  19. Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias • Small sample size • Small number of events • Wide confidence intervals • Uncertainty about magnitude of effect

  20. Methodological limitations Inconsistency of results Indirectness of evidence Imprecision of results Publication bias • Should always be suspected • Only small “positive” studies • Negative or inconclusive studies not published (or delayed) • For profit interest • Outcome reporting bias: Outcomes in a trial reported selectively depending on strength and direction of results

  21. What can raise the quality of evidence? • Large effect • Evidence of dose-response gradient • All plausible confounding would reduce a demonstrated effect or increase the effect if no effect was observed For example: • Observational study data could produce moderate or even high quality evidence if rated up for these factors (no longer providing low quality evidence as initially assumed) • RCT data rated down for e.g. indirectness could be rated up (to remain as high quality evidence)

  22. Schünemann et al. JECH 2010

  23. Quality of evidence Study design High Randomized trial Moderate Low Very low Quality assessment: summary Lower if… Higher if… Study limitations Large effect (e.g., RR 0.5) Very large effect (e.g., RR 0.2) Inconsistency Evidence of dose-response gradient Observational study Indirectness All plausible confounding would reduce a demonstrated effect Imprecision Publication bias 24

  24. Conceptualizing quality • HIGH:We are very confidentthat the true effect lies close to that of the estimate of the effect. • MODERATE: We are moderately confidentin the estimate of effect: The true effect is likely to be close to the estimate of effect, but possibility to be substantially different. • LOW: Our confidence in the effect is limited: The true effect may be substantially different from the estimate of the effect. • VERY LOW: We have very little confidencein the effect estimate: Any estimate of effect is very uncertain. 25

  25. GRADE Evidence Profiles summarize the rating of the quality of evidence across selected outcomes 26

  26. GRADE Evidence Profile Anglemyer A, Rutherford GW, Baggaley RC, Egger M, Siegfried N. Antiretroviral therapy for prevention of HIV transmission in HIV-discordant couples. Cochrane Database Syst Rev 2011: 5 (Aug 2011) 27

  27. GRADE Evidence Profile Anglemyer A et al 2011 28

  28. Quality of evidence reflects not discrete categories but a continuum • Some of the decisions to downgrade or upgrade quality can be borderline • It is the overall evaluation of all the quality issues that should drive these decisions • One should be transparent and acknowledge borderline decisions  GRADE does not ensure reproducible judgments (especially for close-call situations) but requires explicit judgment 29

  29. Recommendation • A recommendation should provide a clear and specific answer to a specific question raised by the target users • Specific questions are formulated as PICO questions: • P: population • I: intervention • C: comparator • O: outcomes 30

  30. Structure of a recommendation • A statement (addressing the elements of PICO) • A grade of the strength of recommendation • A rating of the quality of supporting evidence 31

  31. Strength of recommendation • The extent to which we can be confident that the desirable effects of an intervention outweigh the undesirable effects. 32

  32. Strength of recommendation • A recommendation can be Strong or Conditional: • Strong: panel is confident that the desirable effects of adherence to the recommendation outweigh the undesirable effects (or vice versa). • Conditional: panel concludes that the desirable effects of adherence to the recommendation probably outweigh the undesirable effects (or vice versa), but is not confident.

  33. Types of recommendations • A recommendation can have one of 2 directions: • In favor • Against • As a result there are 4 combinations of strength and direction for a recommendation: • Strong in favorof the intervention • Conditional in favorof the intervention • Conditional againstthe intervention • Strong againstthe intervention

  34. Implications of strong and conditional recommendations 35

  35. Implications of strong and conditional recommendations 36

  36. Implications of strong and conditional recommendations 37

  37. Determining strength of recommendation • Quality of evidence • Balance of benefits and harms • Values and preferences • Resource use • Feasibility

  38. Quality of evidence • The quality of evidence reflects the extent to which confidence in an estimate of the effect is adequate to support recommendations. 39

  39. Quality of evidence • The higher the quality of evidence the more likely the recommendation will be a strong one • The lower the quality of evidence the more likely the recommendation will be a conditional one 40

  40. Balance of benefits & harms 41

  41. Balance of benefits & harms • The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. • The smaller the net benefit and the lower certainty for that benefit, the more likely is a conditional recommendation warranted. 42

  42. Values and preferences • The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely a conditional recommendation is warranted. • In certain cases, certain values and preferences might lead to strong recommendations 43

  43. Resource use • Most of the interventions have resource implications : type, availability, amount • Many of the resource implications are major • Cost 44

  44. Resource use • The higher the costs of an intervention – that is, the more resources consumed – the less likely that a strong recommendation is warranted 45

  45. Risk benefit tables • Summarize the factors that affect the strength of recommendation: • Quality of evidence • Balance of benefits and harms • Values and preferences • Resource use 46

  46. Risk-Benefit Table (example from 2010 MSM Guidelines development process)

  47. Risk-Benefit Table, cont.

  48. Final notes • Conditional recommendations reflect either: • Low quality evidence  need to push the research agenda • Balance of benefits and harms  need to be weighted • Differing values and preferences  need to consider the specific setting 49

  49. Final notes • GRADE does not really introduce new concepts • Emphasizes: • Explicit and structured approach • Transparency • Does not prevent disagreement • Judgments are required • Experts in the field are in the best position to make these judgments • Disagreements need to be explored and built on 50

More Related