Grading evidence and formulating recommendations The GRADE Working Group

1. Grading evidence and formulating recommendations The GRADE Working Group Yngve Falck-Ytter German Cochrane Centre Freiburg, May 7, 2003

2. Background • Need to know: How much confidence in recommendation • Many different grading schemes exists, e.g. • Canadian Task Force on Periodic Health Examinations • United States Preventive Services Task Force • Oxford Center of Evidence Based Medicine • Scottish Intercollegiate Guidelines Network • American Heart Association • The same grading scheme is often used differently • No sufficient explanation how to approach the grading

3. Example Studies on therapy Recom-mendation Level of evidence I RCTs A II Cohort studies B III Case-control-studies IV Case-series C V Expert opinion D Oxford Centre of Evidence Based Medicine; http://www.cebm.net

4. Goal • Simple • Sensible • Explicit • Reliable • Address shortcoming of other schemes(e.g explicitly include trade offs between benefit and risks)

5. Conceptual definitions • Quality of evidence: extent to which one can be confident that an estimate of effect is correct. • Grade (formulation) of a recommendation: extent to which one can be confident that adherence to the recommendation will do more good than harm

6. General Approach 5 step sequential judgment: • Quality of evidence across studies for each important outcome • Which outcomes are critical to a decision • The overall quality of evidence across these critical outcomes • The balance between benefits and harm • Recommendation

7. 1. Quality of evidence of outcome • Study design • RCT vs observational design (not MA, SR, expert) • Study quality • Critical appraisal (outcome dependent) • Consistency • Similarity of effect across studies • Directness • People, intervention & outcome similar to those of interest • Other • Sparse data, high risk of reporting bias

8. Observational studies Quality of evidence Randomized trials Extremely strong association and no major threats to validity High No serious flaws in study quality Strong, consistent association and no plausible confounders Moderate Serious flaws in design or execution or quasi-randomized trials No serious flaws in study quality Low Very serious flaws in design or execution Serious flaws in design and execution Very low Very serious flaws and at least one other serious threat to validity Quality of evidence across outcomes

9. ...continued • Additional factors that lower quality are: • Important inconsistency of results • Uncertainty about directness • Some uncertainty • Major uncertainty can lower the quality by two levels • High probability of reporting bias • Sparse data • Additional factors that may increase quality are: • All plausible residual confounding, if present, would already have reduced the observed effect • Evidence of a dose-response gradient

10. 2. Outcomes critical to a decision • Outcomes may be important, but sometimes not critical to a decision • Only outcomes critical to a decision should provide a basis for recommendation • If information on harm is critical, it should be included even if uncertainty exists

11. 3. Overall quality of evidence • The lowest quality of evidence for any critical outcome should provide the basis for grading • However, if evidence favors the same alternative and there is high quality for some but not all of those outcomes, overall quality should still be high • Weak evidence about implausible putative harms should not lower the overall grade of evidence

12. 4. The balance between benefits and harm • Net benefits: Clearly does more good than harm • Trade-offs: Important trade-offs between benefits and harm • Uncertain trade-offs: It is not clear whether the intervention does more good than harm • No net benefits: Clearly does not do more good than harm

13. 5. Grades of recommendation • „Do it“ or „Don‘t do it“ • Indicating a judgment that a majority of well informed people will make the same choice • Medical practice is expected to not to vary much • „Probably do it“ or „Probably don‘t do it“ • Indicating a judgment that a majority of well informed people will make the same choice, but a substantial minority will not • Medical practice is expected to vary to some degree