How to read a Systematic Review: The FAST tool

1. How to read a Systematic Review: The FAST tool Find Appraise Synthesise Transferability Paul Glasziou Centre for Evidence Based Medicine University of Oxford www.cebm.net

2. The best evidence for different types of question

3. Are RCTs always needed for treatment questions? • Some immediate & dramatic effects don’t need RCTs* • Example: • Child with nasal foreign body • Dislodged with Parent Kiss method • Case series of success 15/19 • Botma J Laryngol Otol 2000 * Glasziou, Chalmers, Rawlins, McCulloch BMJ 2007

4. What do you do? For an acutely ill patient, you do a search • You find 17 studies: • some show significant results; most do not

5. The 17 studies : Forest Plot/Blobbogram • Which is the smallest study? • Which is the largest study? • How many are statistically significant? • Which studies are “large enough”?

6. The 17 studies: of streptokinase for MI

7. How large should the study be?

8. What sample size is needed? For E.H.D.* the usual mortality rate is 0% What sample size is needed to detect a reduction in mortality? • 100 • 1,000 • 100,000 • 1,000,000 * Excessive happiness disorder

9. Sample Size: Café Rule 1The 50:50 Rule (proportions) 50 events are needed in the control group: (For an 80% chance of finding a 50% reduction) Glasziou P, Doll H. Was the study big enough? Two cafe rules. Evid Based Med. 2006;11(3):69-70.

10. What sample size is needed? • There is usually a 12% mortality rate • You think your treatment will lower mortality by 50% • What sample size is needed?

11. What sample size is needed? • There is usually a 12% mortality rate • You think your treatment will lower mortality by 50% • What sample size is needed? • 12% means • 12/100 or 24/200 or 48/400 • and 50 per 417 • Control + Treatment Groups = 834 in total

12. Systematic Review or meta-analysis? • A Systematic Review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. • Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.

13. Is the review any good?FAST appraisal • Question – What is the PICO? • Finding • Did they find most studies? • Appraisal • Did they select good ones? • Synthesis • What to they all mean? • Transferability of results

14. FIND APPRAISE SYNTHESISE TRANSFERABLE Why do I need to check the review? Most reviews do not pass minimum criteria A study of 158 reviews* • Only 2 met all 10 criteria • Median was only 1 of 10 criteria met FAST tool = 4 criteria * McAlister Annals of Intern Med 1999

15. FIND APPRAISE SYNTHESISE TRANSFERABLE What it the review question (PICO)? • Population • Intervention • Comparison • Outcome(s)

16. Do pedometers increase activity and improve health? • Find: what is your search strategy? • Databases? • Terms? • Other methods? Do yourself then Get neighbour’s help

17. FIND APPRAISE SYNTHESISE TRANSFERABLE FIND: Did they find all Studies? • Check for existing systematic review? • Good initial search • Terms (text and MeSH) • At least 2 Databases: MEDLINE, EMBASE, CINAHL, CCTR, ... • Plus a Secondary search • Check references of relevant papers & reviews and • Find terms (words or MeSH terms) you didn’t use • Search again! (snowballing)

18. FIND APPRAISE SYNTHESISE TRANSFERABLE Is finding all published studies enough? • Negative studies less likely to be published than ‘Positive’ • How does this happen? • Follow-up of 737 studies at Johns Hopkins (Dickersin, JAMA, 1992) • Positive SUBMITTED more than negative (2.5 times)

19. FIND APPRAISE SYNTHESISE TRANSFERABLE Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined Simes, J. Clin Oncol, 86, p1529

20. FIND APPRAISE SYNTHESISE TRANSFERABLE Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined Simes, J. Clin Oncol, 86, p1529

21. FIND APPRAISE SYNTHESISE TRANSFERABLE Which are biased? Which OK? • All studies published in BMJ, Lancet, JAMA or NEJM • All publicly funded studies • All studies with more than 100 patients • All studies conducted in the Northern Hemisphere • All studies registered studies

22. FIND APPRAISE SYNTHESISE TRANSFERABLE Publication Bias: Solution • All trials registered at inception, • The National Clinical Trials Registry: Cancer Trials • National Institutes of Health Inventory of Clinical Trials and Studies • International Registry of Perinatal Trials • Meta-Registry of trial Registries • www.controlled-trials.com

24. FIND APPRAISE SYNTHESISE TRANSFERABLE APPRAISE & select studies Did they select only the good quality studies?

25. FIND APPRAISE SYNTHESISE TRANSFERABLE Miscalculating Number Needed to Treat (NNT) 1. EFFICACY: Would the death penalty for miscalculating an NNT decrease miscalculation? Yes/No 2. ATTITUDE: Should we introduce the death penalty for miscalculation of NNT? Yes/No

26. FIND APPRAISE SYNTHESISE TRANSFERABLE Selective Criticism of EvidenceBiased appraisal increases polarization Lord et al, J Pers Soc Psy, 1979, p2098

27. FIND APPRAISE SYNTHESISE TRANSFERABLE Selective Criticism of Evidence 28 reviewers assessed one “study” results randomly positive or negative (Cog Ther Res, 1977, p161-75)

28. FIND APPRAISE SYNTHESISE TRANSFERABLE Assessment: How can you avoid biased selection of studies? • Assessment and selection should be: Standardized “Objective” OR Blinded to Results * assessment of quality blind to study outcome

29. FIND APPRAISE SYNTHESISE TRANSFERABLE Synthesis: pooling the results

30. FIND APPRAISE SYNTHESISE TRANSFERABLE Meta-analysis (Forest) plot Graphical and tabular summary of studies?

31. FIND APPRAISE SYNTHESISE TRANSFERABLE Transferable? Use in my patients Is the AVERAGE effect similar across studies? • If NO, then WHY? • Study methods (RAMbo - biases) • PICO (Patients, Intervention, …) • If YES, then 2 questions • Effect in different individuals? • Which version of treatment?

32. FIND APPRAISE SYNTHESISE TRANSFERABLE Meta-analysis (Forest) plot Are the results similar across studies? 3 tests • Eyeball” test – do they look they same? • Test of “Null hypothesis” of no variation (p-value) • Proportion of variation not due to chance (I2)

33. Epidemiology of Reviews

34. EBM (quick & dirty) Steps Ask Question Search Appraise Apply Time: 90 seconds < 20 articles This patient survives! Systematic Review Steps Ask Question Search ++++ x 2 Appraise x 2 Synthesize Apply Time: 6 months, team < 2,000 articles This patient is dead ConclusionEBM and Systematic Review Find a systematic review!! (and appraise it FAST)

35. Pros and cons of systematic reviews • Advantages • Larger numbers & power • Robustness across PICOs • Disadvantages • May conclude small biases are real effects

36. Is the review any good?FAST appraisal • Question – What is the PICO? • Finding • Did they find most studies? • Appraisal • Did they select good ones? • Synthesis • What to they all mean? • Transferability of results

37. Combined results

38. FIND APPRAISE SYNTHESISE TRANSFERABLE Using review results: what do I do with my patient? • STUDY: meta-analysis of behavioural interventions for insomnia adults • “.. confirms the efficacy of behavioral interventions for person with chronic insomnia.“ • PROBLEM: No regimens for ‘behavioural intervention’ described • Author asked: “what specific treatment regime (or regimes) would you recommend based on your review?” • Author response: “It was found that cognitive, behavioral and relaxation therapies all in general lead to similar improvements in sleep outcomes---although cognitive approaches might have been a bit better. The references for these studies are found in the article. “ Rx “Behavioural Intervention”

40. Summary: systematic reviews • Advantages • Larger numbers & power • Robustness across PICOs • Disadvantages • May conclude small biases are real effects

41. The results: Are studies similar? • What are the overall results? • Similarity of results • Heterogeneity statistic • Similarity of question (PICO) • Your judgement!

42. Traditional Many questions No search methods No inclusion criteria No combining studies Systematic One question Explicit search Explicit inclusion criteria Combine study results(meta-analysis) Traditional cf systematic reviews

43. Which are (i) statistically significant * and (ii) Clinically significant +? (a*+) (b+) (c*) (d) Minimum clinical Important difference No difference

44. Which studies (presented as Odds Ratio): • Are not statistically significant? • Have < 50 patients in the control arm?

45. Sample Size: Cafe Rule 2 - continuousthe 17/(SD squared) Rule For continuous outcomes, number per arm is 17 / (“worthwhile difference” measured in SDs)

46. In a small randomized double-blind trial of a new treatment for acute myocardial infarction, the mortality in the treated group was half that in the control group, but the difference was not significant. We can conclude that: • The treatment is useless • there is no point in continuing to develop the treatment • the reduction in mortality is so great that we should introduce the treatment immediately • we should keep adding cases to the trial until the Normal test for comparison of two proportions is significant • we should carry out a new trial of much greater size

47. The best evidence for different types of question

48. Has the systematic reviewer done a good job?

49. FIND APPRAISE SYNTHESISE TRANSFERABLE Are the studies equivalent? • Are variations in results between studies consistent with chance? (Test of homogeneity: has low power) • If NO, then WHY? • Variation in study methods (RAMbo - biases) • Variation in PICO (Patients, Intervention, …)

50. Is bed rest ever helpful?A systematic review of trials* • 10 trials of bed rest after spinal puncture • no change in headache with bed rest • Increase in back pain • Protocols in UK neurology units - 80% still recommend bed rest after LPSerpell M, BMJ 1998;316:1709–10 • …evidence of harm available for 17 years preceding... *Allen, Glasziou, Del Mar. Lancet, 1999