EAST GRADE Course 2019: GRADE Overview

1. EAST GRADE Course 2019:GRADE Overview Bryce Robinson MD, MS, FACS, FCCM Associate Professor of Surgery Associate Medical Director, Critical Care Harborview Medical Center Department of Surgery University of Washington brobinso@uw.edu @traumabryce

2. No Disclosures

3. Learning Objectives • Understand the shortcomings of older practice management guideline (PMG) methodologies • Understand the basic work flow of GRADE methodology

4. What are PMGs? • Practice management guidelines (PMGs) are statements that include recommendations intended to optimize patient care that are informed by a systematic review of the evidence and an assessment of the benefits and harms of alternative care options. Institute of Medicine, Clinical Practice Guidelines We Can Trust, 2011

5. Things used to be a lot easier…. • To write a PMG, pre-2012 • Form a group of like personalities and specialties • All encompassing PMG for a large topic • Create many, broad questions • ex) When should we remove cervical collars? • Review Pubmed • Give broad recommendations • ex) Cervical collars should be removed as soon as feasible after trauma

6. The Hierarchy of Literature Quality Clinical Observations Available Data Case-Control Studies Cohort Studies Randomized Trials

7. Literature Quality = Recommendation Strength • Article Classification (I, II III): • Class I: “Prospective, randomized clinical trial” • Class II: “Clinical studies in which data were collected prospectively or retrospective analyses based on clearly reliable data • Class III: “Studies based on retrospectively collected data”

8. Literature Quality = Recommendation Strength • Recommendations classification (1,2,3): • Level 1: The recommendation is convincingly justifiable based on the available scientific data. • Usually based on high quality class I data or strong class II data • Level 2: The recommendation is reasonably justified by the data and strongly support by expert opinion. • Usually supported by class II data or a preponderance of class III data • Level 3: The recommendation is supported by available data but adequate scientific evidence is lacking. Supported by class III data. • This type of recommendation is useful for education purposes. Interim Manual for Clinical Practice Guideline Development. Rockville, Md: Agency for Health Care Policy and Research; May 1991.

9. Literature Quality = Recommendation Strength Level I Rec Class I Data “Strong” Level II Rec Class II Data “Preponderance” Level III Rec Class III Data

10. Problems with the Old Way • No widespread, systematic approach • No transparency • How confident/certain are authors that the true effect lies close to the available estimate? • What’s expert opinion (bias)? • Are the authors “reaching” with recommendations? • No accounting for patient preferences, risk/reward of the therapy, charges… • Long PMGs

11. The Big Issue with Older Methods “WEAK” EVIDENCE WEAK RECOMMENDATION BMJ 2003; 327: 1459-61

12. Goals of a High Quality PMG From the Institute of Medicine (2011): • Based on a systematic review of existing evidence • Developed by a knowledgeable, multidisciplinary panel of experts • Considers important patient subgroups and patient preferences • Based on an explicit and transparent process that minimizes distortions, biases, and COI • Provides a clear explanation between care options and outcomes and provides ratings of both evidence quality and strength of recommendations • It is revised as appropriate when new information warrants modification Institute of Medicine, Clinical Practice Guidelines We Can Trust, 2011

13. GRADE Methodology • GRADE Methodology is: • Grading of • Recommendation, • Assessment • Development • Evaluation

14. Other GRADE Groups • >100 organizations from 19 countries

15. Goal of GRADE Methodology • A method to assess the evidence • Evaluation of the totality of the literature • Expressing the quality of literature • Avoidance of expert opinion • Focus on transparency • Distinction between quality of evidence and strength of recommendation • Actionable statements from the authors

16. Framing the Right Question • The first action of the writing group • Can only write a PMG if literature exists for your questions • PMGs synthesize literature, not create it • No points for great questions, only for those that can be answered • No literature = No recommendation = No PMG

17. PICO Questions • GRADE methodology starts with PIC(O) questions • P: Population • I: Intervention • C: Comparison • O: Outcomes • EAST PMGs, try to stick to 2-4 questions • With multiple outcomes

18. What About Outcomes? • Include all possible outcomes at first • Teams make a large list and then vote • Rank of the outcomes, 1-9 • 7-9: Critical and will be included • 4-6: Important but you may not have time (room) • 1-3: Less important, are included

19. What About Outcomes? Critical Mortality Length of Stay Important Patients perception of pain P,I,C Readmission Weak Wound complication Cost

20. Example of Outcome Determination Patel M et al., J Trauma Acute Care Surg. 2015;78:430-441.

21. A Bad Question “Should splenic angioembolization be performed over splenectomy?”

22. A Good PICO Question Population, Intervention, Comparator, Outcome(s) “In adult patients with blunt splenic trauma managed non-operatively (P), should angioembolization (I) be performed compared to no angioembolization (C) to improve splenic preservation (O)?”

23. PICOS and the Literature • For the reader, the PICOs will determine if this PMG aligns to patient practice • For the writer, the questions guide the literature search • What medical subject headings (MeSH) terms to use • Which database to search (good PMGs go beyond PubMed) • *Recruit a professional librarian Kasotakis G et al., J Trauma Acute Care Surg. 2016;82:618-626.

24. Quantitative Synthesis for Effect Estimates • If able, PMGs will present meta-analyses if multiple studies exist with sufficiently similar “PICO type” architecture (similar conditions with similar subjects) • I2 statistic = • % of variation across studies that is due to heterogeneity rather than chance. • >75% = high • <40% = low

25. Qualitative Synthesis of the Literature Strength • How certain is the author group in an estimate of effect after reviewing the body of literature? From the GRADE working group, Rebecca Morgan

26. Evidence Quality • High = very confident that the true effect lies close to that of the estimate of effect • Moderate = the true effect is likely to be close to the estimate of the effect • Low = confidence in the effect estimate is limited • Very low = very little confidence in the effect estimate

27. Lowering Certainty of the Effect • For each OUTCOME of the PICO • When to down-score the literature quality? • Risk of bias = randomization issues, concealed allocation, blinding, completeness of follow up… • Inconsistency = heterogeneity of the papers, unexplained different results between trials • Indirectness = different P,I and Os from the PICOs • Imprecision = low event rate, wide confidence intervals • Publication bias = the rare negative study, ‘gray’ literature

28. Raising the Certainty of the Effect • For each OUTCOME of the PICO • When to up-score the quality of the literature? • Mainly for observational trials • Large effect size (large risk ratios 0.5 or 2.0) • Dose-response exists • Plausible residual confounding; significant effect seen even in presence of major confounding

29. The Evidence Profile • After the quantitative (meta analysis) and qualitative grading of the literature quality, it’s time to make an evidence profile table. • An evidence profile is universal to GRADE methodology • The table represents: • A single PIC and many Os • All of the OUTCOMEs for that particular question • Provides the “quality assessment” (reason for altering certainty) for each OUTCOME • Provides a “summary of findings” (data) for each OUTCOME • The overall quality of the literature for the PIC(O) is based on the critical OUTCOME with the LOWEST quality

30. Example of an Evidence Profile Fox N et al., J Trauma Acute Care Surg. 2015;78:136-146.

31. Putting It All Together for Each PICO Question Overall Evidence Quality over all Outcomes Balance of Benefits vs. Harms Feasibility, Equity, and Acceptability RECOMMENDATION Patient’s Values and Preferences Resource Use Expert Opinion

32. The Strength of Recommendations • The “strength” of the recommendation reflects the extent to which we can be confident that the desirable effects of an intervention outweigh the undesirable effects. • Strong vs. weak recommendations

33. The Strength of Recommendations • A strong recommendation for: • Patients means: most and only a small proportion would not want the recommended therapy/intervention • Clinicians means: most patients should receive the recommended therapy/intervention • Policymakers means: that the recommendation can be evaluated as a new policy or performance measure • This is the new “standard of care”

34. The Strength of Recommendations • A weak (conditional) recommendation: • Used when the desirable effects of adherence to a recommendation are uncertain. • May be due to low-quality evidence (most common) • May be because the desirable and undesirable effects are closely balanced • May be due to a wide variation in patient preferences • May be due to a large resource need to implement the recommendation • “We conditionally recommend for/against” or “we suggest for/against”

35. The Strength of Recommendations • A weak recommendation for: • Patients means: most would want the recommended therapy/intervention but many would not • Clinicians means: that different choices with be appropriate for different patients • Policymakers means: that the recommendation will require more debate and involvement of stakeholders before policy can be determined

36. Summary of GRADE From the GRADE working group, Shahnaz Sultan

37. QUESTIONS?

38. Learning Objectives • Understand the shortcomings of older practice management guideline (PMG) methodologies • Understand the basic work flow of GRADE methodology

39. Thank You