Why Grade the Evidence?

1. Why Grade the Evidence? • target audience for Cochrane reviews • clinicians interested in the question • policy makers, consumers • many in audience will have limitations • time • methodological sophistication

2. Implications of Limitations • limited time/sophistication • is evidence strong, inferences secure? • is evidence weak, inferences insecure? • shorthand summary • enhance usefulness of reviews • Gray Elrodt in the Berkshires • post-MI smoking cessation, aspirin, beta blocker, ACE inhibitor, statin

3. Grade for Specific Question • too vague: alendronate in osteoporosis • patients – post-menopausal women • intervention • daily alendronate, dose 10 to 20 mg. • outcome – non-vertebral fractures

4. What Influences Grade? • study design • basic • detailed design and execution • consistency • directness • reporting bias

5. Summary Methodological Quality • study design • randomization • observational study • detailed design and execution • concealment • balance in known prognostic factors • intention to treat principle observed • blinding • completeness of follow-up

6. Summary Methodologic Quality • consistency of results • if inconsistency, look for explanation • patients, intervention, outcome, methods • no clear threshold • size of effect, confidence intervals, statistical significance

7. Risk Difference of Conversion to Sinus Rhythm Amiodarone vs Placebo or Digoxin or CCB Favours Control Favours Amiodarone AF Duration > 48 hrs ' Bianconi 2000 0.02 (-0.06 to 0.11) ' Galperin 2000 0.33 (0.19 to 0.47) ' Hohnloser 2000 0.16 (0.06 to 0.26) ' Natale 2000 0.67 (0.51 to 0.83) ' Villani 2000 0.20 (0.06 to 0.33) AF Duration </= 48 hrs ' Cowan 1986 0.08 (-0.19 to 0.36) ' Noc 1990 0.77 (0.52 to 1.00) ' Capucci 1992 -0.11 (-0.41 to 0.20) ' Cochrane 1994 0.13 (-0.06 to 0.33) ' Donovan 1995 0.03 (-0.21to 0.27) ' Hou 1995 0.21 (-0.01 to 0.43) ' Kondili 1995 0.14 (-0.16 to 0.44) ' Galve 1996 0.08 (-0.11 to 0.27) ' Kontoyannis 1998 0.31 (0.11 to 0.50) ' Bellandi 1999 0.27 (0.13 to 0.41) ' Cotter 1999 0.28 (0.13 to 0.43) ' Kochiadakis 1999 0.28 (0.15 to 0.41) ' Joseph 2000 0.19 (-0.02 to 0.39) ' Peukurinen 2000 0.52 (0.31 to 0.72) ' Vardas 2000 0.41 (0.28 to 0.53) ' Cybulski 2001 0.39 (0.24 to 0.54) ' Pooled Risk Difference 0.26 (0.18 to 0.34) -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2

8. Relative Risk of Hospital Mortality: Adult Patients Favours Favours Private Private For-Profit Not-For-Profit Number of Number of Study % Weight Hospitals Patients ! Shortell 653 144,159 1.43 ! Keeler 220 4,937 0.04 ! Hartz 2,368 3,107,616 11.38 ! Manheim MH 1,252 1,537,660 9.78 ! Manheim FS 1,617 2,228,593 2.59 ! Kuhn 2,580 3,353,676 12.34 ! Pitterle 3,482 4,529,206 14.11 ! Mukamel 1,653 5,298,812 17.21 ! Bond 3,224 4,210,468 12.66 ! Yuan Medical 3,316 7,386,000 11.90 ! Yuan Surgical -- 4,396,000 5.05 ! Lanska 799 16,983 0.00 ! McClellan 2,875 181,369 1.48 ! Sloan 2,360 7,079 0.03 Totals 26,399 36,402,558 100.00 ! Random Effects Pooled Estimate 0.7 0.8 0.9 1 1.1 1.2 1.3 Relative Risk and 95% CI

9. Directness of Evidence • indirect treatment comparisons • interested in A versus B • have A versus C and B versus C • alendronate vs risedronate • both versus placebo, no head-to-head

10. Four Levels of “Directness” • patients meet trials’ eligibility criteria • not included, but no reason to question • slight age difference, comorbidity, race • some question, bottom line applicable • valvular atrial fibrillation • serious question about biology • heart failure trials applicability to aortic stenosis

11. Levels of Directness • interventions • same drugs and doses • similar drugs and doses • same class and biology • questionable class and biology • outcomes • same outcomes • similar (duration, quality of life) • less breathlessness for role function • laboratory exercise capacity for q of life

12. Magnitude, Precision, Reporting Bias • magnitude not generally part of quality • but very large magnitude can upgrade • precision not generally part of quality • but sparse data can lower quality • reporting bias • high likelihood can lower quality

13. Grading System • high quality well done RCT • intermediate quasi-RCT • low well done observational • insufficient anything else

14. Moving Down • study execution • serious flaws can lower by one level • fatal flaws can lower by two levels • consistency • important inconsistency can lower by one level • directness of evidence • some uncertainty re relevance lower by one level • major uncertainty re relevance lower by two levels • selection bias • strong evidence lower by 1 level

15. Moving Up • very strong association, up 2 levels • insulin in diabetic ketoacidosis • strong, consistent association with no plausible confounders, up 2 levels • fluoride for preventing cavities • strong association can move up 1 level • ? HRT ?

16. Conclusion • challenges in grading • balancing simplicity and complexity • judgement always required • consistent grading system required • must consider study design, execution, consistency, directness • magnitude, precision, publication bias only when extreme

17. Questions for Discussion • should Cochrane reviews provide: • grades of evidence across studies for all important outcomes? • overall grade of evidence across outcomes? • same grading system across reviews? • bonus; while Cochrane reviews shouldn’t provide recommendations should they include structured discussion including • trade-offs • translation of evidence in to practice