Critical Appraisal of Systematic Reviews

1. Critical Appraisal of Systematic Reviews Douglas Newberry

2. Systematic Reviews — or How to make a Monkey out of EBM without hardly trying!

3. Systematic Reviews:Objectives: Appraise a systematic review for validity Discuss Meta Analysis / use Odds Ratios Obtain Number Needed to Treat (NNT) from Odds Ratios Consider clinical implications of a Systematic Review {including when to bin it instead!}

4. We can see further than our forbearers because we stand on the shoulders of Giants{and have better spectacles} • these ideas are cribbed unashamedly from friends, books & previous courses

5. Systematic Reviews:What are your Objectives: What do you want to cover? Please interject with helpful questions!

6. Did I really want a systematic review?(but please do not pretend) • admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence). • if the question is important > formulate it! • Systematic review > narrow but rigorous focus.

7. Systematic Reviews — Where do I start: • Start with your 4 (or 3) part clinical question! • Is a systematic review a sensible approach? • Does THIS systematic review address MY question? • Is it a systematic review at all?

8. Is it a systematic review? does it: • define a four part (answerable) clinical question? • combine Randomized Controlled Trials (RCT’s)? • describe PRE-DEFINED search methods? • PRE-DEFINED inclusion criteria? • PRE-DEFINED methodological exclusion criteria?

9. Sceptical View? Take it with a grain of salt: • transparent declaration of funding of work? • Drug Company sponsorship of Reviews vs. Methodological quality>Cochrane review! • who employs the authors? • open discussion of existing controversy & commercial gain? • Don’t waste salt on your food, keep it for your reading!

10. Meta analysis — combine what with what? • Low Molecular Weight Heparin (LMWH) in hip surgery — begin before or after the operation? • meta analysis of placebo controlled RCT’s of heparin in hip surgery >> • pre-op & post-op LMWH vs. placebo • post-op LMWH Vs placebo • pre-operative >> less intra-op bleeding??

11. Can we believe it ? • bias free search & inclusion criteria? • appraisal of methodology of primary studies? • consistent results from all primary studies? • if not, are the differences sensibly explained? • are the conclusions supported by the data?

12. If we believe it — does it apply to our patient? • Is our patient (or population) so different from those in the primary studies that the results may not apply? • consider differences in: • time — many things change. • culture — both treatments and values of outcomes can be different • stage of illness or prevalence can effect results.

13. We believe it ! but—>> does it matter? • Is the benefit worthwhile to our patient? • Ask the patient about cultural values. • Think about Relative Risk Reduction vs. Absolute Risk to our patient. • Potential benefit is the Absolute risk avoided in our patient = Absolute Risk Reduction (ARR)!

14. Absolute Risk—> The risk our patient is facing! • How likely is our patient to die (or have the outcome of interest) without intervention? = Control Event Rate (CER) • consider this individual patient’s risk factors to estimate Patient Expected Event Rate = PEER. • Absolute Risk usually increases with age. • Improvement measured as Absolute Risk Reduction (ARR)

15. Relative Risk Reduction: • Usually reported in studies. • Ratio of the improvement of outcome over outcome without intervention (Rx): • {Control Event Rate (CER) — Experimental Event Rate (EER)} / CER • i.e. {CER-EER}/CER • often independent of prevalence! • often similar at different ages!

16. Our patient wants an absolute Risk Reduction (ARR): • is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote! • if I have a 30% risk of MI or death {30 out of 100 people like me will suffer MI or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death. ARR = 12 out of 100! >>I like that! • BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different!

17. Number Needed to Treat (NNT) (very trendy but tricky): • only defined for specific prevalence-Patient’s Expected Event Rate=PEER! • only defined for a specific intervention! • only defined for a specific outcome! • eg. Pravastatin™ 40 mg nocte x10 years, in a 65 year old male, ex-smoker with high BP and Diabetes, to reduce MI or Death. • NNT is the inverse of Absolute Risk Reduction: i.e. NNT = 1/ARR

18. Number Needed to Treat (NNT) for previous example: • 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100 • NNT = 1/(12/100)=100/12= 8.3 • But the same Relative Risk Reduction (RRR) of 40% with a low prevalence: • 0.4 fewer MI/death per 100 treated, ARR=0.4/100. • NNT = 1/(0.4/100) = 100/0.4 = 250!

19. Why Odds Ratios? > compare results of different studies. • consider 2x2 table: • RRR is (a-b/a) — but you can only go in rows within same study! • Odds ratio is (a/c)/(b/d) = ad / bc — the individual ratios are in columns, and therefore are independent of the prevalence which is different in different studies. • must use odds ratios to combine RCT’s

20. Odds Ratio (OR) to NNT — is the improvement worth the trouble? • 1>OR>0, lower the OR = better the treatment (Rx) >> lower NNT. • for any OR, NNT is lowest when PEER=0.5 • estimate the PEER (patient’s risk) • apply the OR to get patient's NNT.

21. Convert PEER & OR to NNT:

22. Formula used in the table:

23. Table induced nausea! • lower OR >> lower NNT • Patient needs to be at risk (non-trivial PEER) in order for risk reduction to be worth the effort. • for any OR, NNT lowest when PEER=0.5 • more effective treatment > lower NNT • BUT are your patient’s values satisfied by the intervention and its sequelae?

24. Subgroup analysis: Sceptical unless: • the subgroups make biological and clinical sense? • the differences are both clinically & statistically significant? • was a-priori hypothesis (before this data)? • other evidence supports these sub-groups? • few (OK) or many (nix) sub-group analyses?

25. Any Questions?

26. Summary 1: Set your goals. • define your 4 (or 3) part question. • do you want a true systematic review? • does this narrow review address my question? • PRE-DEFINED search, inclusion, exclusion!

27. Summary 2: Be Sceptical! • look for bias, conflict of interest. • critical appraisal of primary studies? • consistent results? if not, why not? • does our patient fit the groups studied? • does it matter to our patient?

28. Summary 3: Risks that matter. • Absolute risk > estimate the Patient Expected Event Rate (PEER) • obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis • plug into a table to estimate Number Needed to Treat (NNT)

29. Summary 4: Sceptical & common sense! • beware of post-hoc sub-group analysis, especially if multiple. • step back and consider if the systematic review really related to our patient’s situation (PEER), culture and expectations? • do not loose sight of common sense!

30. Coffee Now! • Small Groups Afterwards