The GRADE system

1. Holger Schünemann, MD, PhD Chair, Department of Clinical Epidemiology & Biostatistics Professor of Clinical Epidemiology, Biostatistics and Medicine McMaster University, Hamilton, Ontario, Canada The GRADE system

2. 1. Formulating questions Guidelines are a way of answering questions about clinical, communication, organisational or policy interventions, in the hope of improving health care or health policy. It is therefore helpful to structure a guideline in terms of answerable questions. WHO Guideline Handbook, 2008

3. Different types of questions Background Questions Definition: e.g. What is Human Papilloma Virus (HPV) infection?  Mechanism: e.g. How does HPV cause cancer? Foreground Questions Efficacy: e.g. What is the efficacy of an HPV vaccine? Recommendations/decisions: e.g. e.g. Should we use HPV vaccine?

4. Different types of questions Background Questions Definition: e.g. What is Human Papilloma Virus (HPV) infection?  Mechanism: e.g. How does HPV cause cancer? Foreground Questions Efficacy: e.g. What is the efficacy of an HPV vaccine? Recommendations/decisions: e.g. e.g. Should we use HPV vaccine? Actionable items

5. 2. Choosing outcomes • Every decision comes with desirable and undesirable consequences • Developing recommendations must include a consideration of desirable and undesirable consequences

6. Desirable and undesirable consequences • desirable effects • lower mortality • improvement in quality of life, fewer hospitalizations • reduction in the burden of treatment • reduced resource expenditure • undesirable consequences • deleterious impact on morbidity, mortality or quality of life, increased resource expenditure

7. Limitations of older systems & approaches • confuse quality of evidence with strength of recommendations • lack well-articulated conceptual framework • criteria not comprehensive or transparent • focus on single outcomes

8. Grades of Recommendation Assessment, Development and Evaluation GRADE Working Group CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008, Lancet ID 2007, PLOS Medicine 2007

9. GRADE Working Group David Atkins, chief medical officera Dana Best, assistant professorb Martin Eccles, professord Francoise Cluzeau, lecturerx Yngve Falck-Ytter, associate directore Signe Flottorp, researcherf Gordon H Guyatt, professorg Robin T Harbour, quality and information director h Margaret C Haugh, methodologisti David Henry, professorj Suzanne Hill, senior lecturerj Roman Jaeschke, clinical professork Regina Kunx, Associate Professor Gillian Leng, guidelines programme directorl Alessandro Liberati, professorm Nicola Magrini, directorn James Mason, professord Philippa Middleton, honorary research fellowo JacekMrukowicz, executive directorp Dianne O’Connell, senior epidemiologistq Andrew D Oxman, directorf Bob Phillips, associate fellowr Holger J Schünemann, professorg,s Tessa Tan-Torres Edejer, medical officert David Tovey, Editory Jane Thomas, Lecturer, UK Helena Varonen, associate editoru Gunn E Vist, researcherf John W Williams Jr, professorv Stephanie Zaza, project directorw a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USA c) Centers for Disease Control and Prevention, USA d) University of Newcastle upon Tyne, UK e) German Cochrane Centre, Germany f) Norwegian Centre for Health Services, Norway g) McMaster University, Canada h) Scottish Intercollegiate Guidelines Network, UK i) FédérationNationale des Centres de LutteContre le Cancer, France j) University of Newcastle, Australia k) McMaster University, Canada l) National Institute for Clinical Excellence, UK m) Università di Modena e Reggio Emilia, Italy n) Centro per la ValutazionedellaEfficaciadellaAssistenza Sanitaria, Italy o) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Poland q) The Cancer Council, Australia r) Centre for Evidence-based Medicine, UK s) National Cancer Institute, Italy t) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USA x) University of London, UK Y) BMJ Clinical Evidence, UK

10. The GRADE approach Clear separation of 2 issues: 1) 4 categories of quality of evidence: very low, low, moderate, or high quality? • methodological quality of evidence • likelihood of systematic deviation from truth • by outcome 2) Recommendation: 2 grades – conditional or strong (for or against)? • Quality of evidence only one factor *www.GradeWorkingGroup.org

11. Determinants of quality • RCTs start high • observational studies start low • 5 factors lower the quality of evidence • limitations in detailed design and execution • inconsistency • indirectness • reporting bias • imprecision • 3 factors can increase the quality of evidence

12. Example: Limitations in Design and Execution • Limitations – observational studies • Failure to develop and apply appropriate eligibility criteria - under- or over-matching in case-control studies • Selection of exposed and unexposed in cohort studies from different populations • Flawed measurement of both exposure and outcome (e.g. recall bias in CC studies) • Differential surveillance for outcome in exposed and unexposed in cohort studies • Failure to adequately measure/control for confounding • Failure to match for prognostic factors and/or adjustment in statistical analysis

13. Quality assessment criteria

14. From Evidence to Recommendation

15. Categories of recommendations Although the degree of confidence is a continuum, we suggest using two categories: strong and weak/conditional. Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects. Conditional recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident. Recommend   Suggest  

16. Determinants of the strength of recommendation

17. Judgements about the strength of a recommendation No precise threshold for going from a strong to a weak recommendation The presence of important concerns about one or more of these factors make a weak recommendation more likely. Panels should consider all of these factors and make the reasons for their judgements explicit. Recommendations should specify the perspective that is taken (e.g. individual patient, health system) and which outcomes were considered (including which, if any costs).

18. Finally: There are no RCTs! • We will do a consensus statement/guideline (and not use rigorous methods) • Do you think that those using the recommendations would like to be informed about the basis (explanation) for a recommendation if they were asked (by their patients)? • I suspect the answer is “yes”

19. There are no RCTs! Cont’d • Another reason for using structured approaches: any form of recommendation needs agreement/consensus – whether based on high or lower quality evidence (voting as a forced form of consensus) • Ergo: Consensus statement is a misnomer in regards to differentiating from guideline • The level of detail depends on other aspects: • Funds, time, greater interest, higher priority • Transparency is key

20. Conclusions • Clinical practice guidelines should be based on the best availableevidence • GRADE provides a structure approach to improve communication – official WHO system • Criteria for evidence assessment across questions and outcomes • Criteria for moving from evidence to recommendations • Transparent, systematic • four categories of quality of evidence • two grades for strength of recommendations • Transparency in decision making and judgments is key

22. What can raise quality? 2. dose response relation • (higher INR – increased bleeding) • childhood lymphoblastic leukemia • risk for CNS malignancies 15 years after cranial irradiation • no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%) 3. all plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

23. All plausible confounding would result in an underestimate of the treatment effect • Higher death rates in private for-profit versus private not-for-profit hospitals • patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals • for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect)

24. All plausible biases would result in an overestimate of effect • Hypoglycaemic drug phenformin causes lactic acidosis • The related agent metformin is under suspicion for the same toxicity. • Large observational studies have failed to demonstrate an association • Clinicians would be more alert to lactic acidosis in the presence of the agent

26. Implications of a strong recommendation Patients: Most people in your situation would want the recommended course of action and only a small proportion would not Clinicians: Most patients should receive the recommended course of action Policy makers: The recommendation can be adapted as a policy in most situations

27. Implications of a weak/conditional recommendation Patients: The majority of people in your situation would want the recommended course of action, but many would not Clinicians: Be prepared to help patients to make a decision that is consistent with their own values Policy makers: There is a need for substantial debate and involvement of stakeholders

28. Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)?

29. Should oseltamivir be used for treatment of patients hospitalised with avian influenza (H5N1)? Summary of findings Transmission: No human to human transmission Patient or population: Hospitalised, clinical and serologically confirmed cases of avian influenza

30. Oseltamivir for Avian Flu Summary of findings: No clinical trial of oseltamivir for treatment of H5N1 patients. 4 systematic reviews and health technology assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. Hospitalization: OR 0.22 (0.02 – 2.16) Pneumonia: OR 0.15 (0.03 - 0.69) 3 published case series. Many in vitro and animal studies. No alternative that is more promising at present. Cost: ~ Euro 50 per treatment course

31. What would you recommend? • Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects. • Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.

32. Judgments about the strength of a recommendation - oseltamivir for treatment of patients hospitalised with avian influenza (H5N1)

33. Who would recommend oseltamivir for these patients (no other alternative)? • YES (green card) • No (pink card)

34. Example: Oseltamivir for Avian Flu Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence). Schunemann et al., The Lancet ID, 2007

35. Example: Oseltamivir for Avian Flu Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence). Values and Preferences Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment. Schunemann et al., The Lancet ID, 2007

36. Other explanations Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time. The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote. Schunemann et al., The Lancet ID, 2007