How GRADE could help to implement the evidence

1. Holger Schünemann, MD, PhD Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics Professor of Medicine Michael Gent Chair in Healthcare Research McMaster University, Hamilton, Canada How GRADE could help to implement the evidence

2. Content Systematic Reviews & GRADE Evidence & judgments Recommendation, health policy & implementation

3. Content • Examples and summary from leading/co-leading 15 guideline projects • 11 World Health Organization • World Allergy Organization, Allergic Rhinitis in Asthma, American Thoracic Society (2) Systematic Reviews & GRADE Evidence & judgments Recommendation, health policy & implementation

4. Guideline development Process

5. Create evidence profile with GRADEpro Summary of findings & estimate of effect for each outcome Guideline development Grade overall quality of evidence across outcomes based on lowest quality of critical outcomes Rate quality of evidence for each outcome Outcomes across studies Randomization increases initial quality Formulate question Rate importance Select outcomes Risk of bias Inconsistency Indirectness Imprecision Publication bias P I C O Outcome Critical High Outcome Critical Moderate Grade down Low Outcome Important Very low Outcome Not important Large effect Dose response Confounders Grade up Panel • Formulate recommendations: • For or against (direction) • Strong or conditional/weak (strength) • By considering: • Quality of evidence • Balance benefits/harms • Values and preferences • Revise if necessary by considering: • Resource use (cost) Systematic review • “We recommend using…” • “We suggest using…” • “We recommend against using…” • “We suggest against using…”

6. Outcome generation and selection • Multidisciplinary panels • Researchers, epidemiologists, public health officers, methodologists, patient representatives… • Delphi process • 3 rounds • List of possible outcomes from literature • Panel members review and add • List of all outcomes grouped by theme • Panel members rate importance • Final agreement and results

8. Outcomes for screening on a scale of 1 (not important) to 9 (critical)

9. Challenges and advantages of this approach • Often starting with many outcomes • Experts initially focused on what they know from research studies • Requires detailed explanations • ↓ Participation of panel members • Perspective taken • Complete • Everyone involved • Independent ratings • Numerical estimates • Well documented and kept record • Transparent • Reduces work

10. WHO influenza guidelines • New guideline on pharmacological management of influenza • Previously few randomized trials • Low quality evidence for many outcomes (imprecision) • Industry sponsored – publication bias • Not all outcomes • Review of observational studies • To inform guidelines

12. Methods • Standard systematic review • MEDLINE, EMBASE, CENTRAL, CINAHL, SIGLE, the Chinese Biomedical Literature Database, Panteleimon and LILACS for relevant studies up to November 2010 • contacted pharmaceutical companies and international agencies • RevMan 5.1 • 10 PICO → recommendations approach • Outcomes determined through Delphi process previously • QoE according to GRADE approach • GRADEpro (www.gradeworkinggroup.org) • Risk of bias using modified Ottawa Newcastle scale

13. Results Question: Should oseltamivir vs. no antiviral treatment be used for influenza (follow-up: 30 days)?

14. Content Systematic Reviews & GRADE Evidence & judgments Recommendation, health policy & implementation Systematic and transparent approach Transparently lay out rationale for recommendations Manage COI

15. Judgments/explanations 1 Although we did not downgrade, publication bias cannot be excluded and is of concern.2 Studies not adjusted for potential confounding factors.3 Significant differences in effect for pandemic versus seasonal influenza (see subgroup analyses table).4 Publication bias a concern since large studies had for-profit funding and weighted heavily in analyses.5 No independent comparison group.6 High heterogeneity among studies.7 Measured in select patients in trials.

16. Getting from evidence to recommendations - GRADE Recommendations are based on judgments: • Quality of evidence (confidence in estimates of effect) • Balance between benefits and downsides • Values and preferences • Resource use But judgments need to be based on the best available evidence and transparent

17. Balancing desirable and undesirable consequences ↑ Resources ↑ Nausea ↑ herd immunity ↓ Morbidity ↑ Local skin reactions ↑ Allergic reactions ↓ Death ↑ QoL Conditional Strong For Against

18. Balancing desirable and undesirable consequences ↑ Resources ↑ Nausea ↑ herd immunity ↓ Morbidity ↑ Local skin reactions ↑ Allergic reactions ↓ Death ↑ QoL Conditional Strong For Against

19. Balancing desirable and undesirable consequences ↑ Resources ↑ Nausea ↑ herd immunity ↓ Morbidity ↑ Local skin reactions ↑ Allergic reactions ↓ Death ↑ QoL Conditional Strong For Against

20. Balancing desirable and undesirable consequences ↑ Resources ↑ Nausea ↑ herd immunity ↓ Morbidity ↑ Local skin reactions ↑ Allergic reactions ↓ Death ↑ QoL Conditional Strong For Against

21. Balancing desirable and undesirable consequences ↑ Resources ↑ Nausea ↑ herd immunity ↓ Morbidity ↑ Local skin reactions ↑ Allergic reactions ↓ Death ↑ QoL Conditional Strong For Against

22. Content Systematic Reviews & GRADE Evidence & judgments Recommendations, health policy & implementation

24. Depending on contact investigation strategy used, resource utilization and implications will vary. • Opportunity cost may be high. • Feasibility is dependent on existing and well functioning programs. • Resources worth in smear positive index cases

25. Should cryotherapy versus LEEP be used in women with histologically confirmed cervical intraepithelial neoplasia?

26. Recommendation • In settings where LEEP is available and accessible, and women present with CIN lesions extending into the cervical canal, the expert panel suggests treatment with LEEP over cryotherapy (conditional recommendation, OO quality evidence) • Remarks: The benefits of LEEP were greater than those of cryotherapy, and the harms were fewer in these women. However, since there are greater resource implications for LEEP than cryotherapy, and thus LEEP is not available in all settings, a conditional recommendation was made.

27. Implications of a conditional/weak recommendation • Patients: The majority of people in this situation would want the recommended course of action, but many would not • Clinicians: Be more prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making • Policy makers: There is a need for substantial debate and involvement of stakeholders

28. Implications of a strong recommendation • Patients: Most people in this situation would want the recommended course of action and only a small proportion would not • Clinicians: Most patients should receive the recommended course of action • Policy makers: The recommendation can be adapted as a policy in most situations , can be used as quality indicator/performance measure

29. Only two of six performance measures seemed reasonable

30. WHO evaluation & feedback • WHO staff & guideline review committee members (GRC) invited to feedback (Jan 2011) about using GRADE approach • what worked • what did not work • Group discussion (NGT), 11 + 2

31. Summary of feedback • Transparency of the GRADE process helps • Requirement for a good Chair • methods knowhow to move the process • For observational studies we need better/ ¿different? summaries (e.g. narrative versions) • Integration and elicitation of values and preferences was frequently challenging • For global guidelines: issues around the description of baseline risks and applicability across countries require work

32. Summary of feedback • Variability in baseline risk → weak or conditional recommendations should follow • A description of the modifying factors and the layout of the evidence could be a great benefit and will facilitate implementation • Measures should be taken to streamline the timing of the development of guidelines • Working with centers, training, and capacity building of these centers who collaborate with WHO a priority for implementing GRADE • Impact evaluation (of current process for development of guidelines) should take place

33. Conclusions • (WHO) guidelines should be based on the best available evidence to be evidence based • GRADE not avoid judgments but provides framework • combines what is known in health research methodology and provides an approach to improve communication • GRADE process works – is it better? • Change in culture towards the use of evidence • Transparency in decision making and judgments is key

34. Thanks • Nancy Santesso, Andy Oxman, Suzanne Hill • WHO staff who participated in providing feedback

36. Records identified through database searching (all study designs)EMBASE, MEDLINE = 9873 SIGLE = 7 CINAHL = 1062 LILACS = 19 COCHRANE = 301 Chinese Biomedical Literature Database = 914 Panteleimon = 12 (Total n = 12176) Results - PRISMA Additional records identified through other sources Pharmaceutical companies(n = 12) Reference lists of relevant papers (n=15) Records after duplicates removed(n = 7456) Records screened(n = 7483) Records excluded(n = 6563) Studies awaiting assessment (n = 6) • Studies awaiting translation (1) • Papers could not obtain in full (5) Full-text articles assessed for eligibility(n = 920 ) Full-text articles excluded (n = 825) Excluded for • Not influenza or influenza like illness • Fewer than 25 people • Randomised controlled trial, or not an observational study • Not antiviral agent • Antiviral agents analysed together • Prophylaxis • No outcomes reported Studies included N = 89 Question • 51 + 5 studies • 7 studies • 6 studies • 0 studies • 8 • 0 studies • 16 • 0 studies • 1 study • 2 studies Note: one study may be relevant to multiple questions

37. Results Should oseltamivir versus no treatment be used to treat influenza? Mortality (adjusted) Mortality (unadjusted)

38. Recommendation • The Guidelines Group recommends that TB programs/clinicians use/do not use fluoroquinolones in the treatment of all patients with MDR (Strong (conditional) recommendation/ low (very low, low, moderate, high) grade of evidence)

39. Example: Oseltamivir for Avian Flu Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivirtreatment as soon as possible (strong recommendation, very low quality evidence). • Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment. Schunemann et al. The Lancet ID, 2007

40. Other explanations Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time. The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.

43. Methods Types of participants • We included studies in all populations with influenza or influenza like-illness. Types of intervention • Oseltamivir, zanamivir, amantadine or rimantadine in any dose or by any route. Type of outcome measures • We determined a priori to report on the following outcomes because they were judged to be important or critical for decision making: • Mortality,Hospitalisation, ICU Admission, mechanical ventilation and respiratory failure, Duration of hospitalization,Time to alleviation of symptoms, Time to return to normal activity, Complications • Critical adverse events (e.g. major psychotic disorders, encephalitis, stroke and seizure), • Important adverse events (e.g. pain in extremities, clonic twitching, body weakness, dermatological changes such as uticaria and rash) • Viral shedding andResistance

44. GRADE Uptake • World Health Organization • Allergic Rhinitis in Asthma Guidelines (ARIA) • American Thoracic Society • American College of Physicians • European Respiratory Society • European Society of Thoracic Surgeons • British Medical Journal • Infectious Disease Society of America • American College of Chest Physicians • UpToDate® • National Institutes of Health and Clinical Excellence (NICE) • Scottish Intercollegiate Guideline Network (SIGN) • Cochrane Collaboration • Infectious Disease Society of America • Clinical Evidence • Agency for Health Care Research and Quality (AHRQ) • Partner of GIN • Over 60 major organizations

45. GRADE Uptake • World Health Organization • Allergic Rhinitis in Asthma Guidelines (ARIA) • American Thoracic Society • American College of Physicians • European Respiratory Society • European Society of Thoracic Surgeons • British Medical Journal • Infectious Disease Society of America • American College of Chest Physicians • UpToDate® • National Institutes of Health and Clinical Excellence (NICE) • Scottish Intercollegiate Guideline Network (SIGN) • Cochrane Collaboration • Infectious Disease Society of America • Clinical Evidence • Agency for Health Care Research and Quality (AHRQ) • Partner of GIN • Over 60 major organizations

46. Recommendation 1a • The panel recommends that people who had household contact with smear positive or M/XDR TB index cases be investigated for active TB (strong recommendation, very low quality evidence).

47. Recommendation 1b • The panel recommends that people who had household contact with TB index cases who are children younger than 5 years of age be investigated for active TB (strong recommendation, very low quality evidence).