SYSTEMATIC REVIEWS AND META-ANALYSIS

1. SYSTEMATIC REVIEWS AND META-ANALYSIS

2. Objectives • Define systematic review and meta-analysis • Know how to • access • appraise • interpret the results of a systematic review

3. Reasons to use reviews • Sheer volume of literature* • Save time doing exhaustive literature researches • *Wyatt: about 25,000 biomedical journals in print worldwide. • This number is rising by 4% pa - doubling time of 19 yrs

4. Review • Any attempt to synthesise results and conclusions of 2 or more publications on a given topic • e.g. editorials, working papers • Problems • Retrieval bias and publication bias • Were all studies identified? • Do not see how conclusions arrived at • Reviewers may be biased / conflict of interest • Languages excluded?

5. Were all studies that met the inclusion criteria for the review identified successfully? If not, were the results of the sample of studies included in the review representative of the results of all eligible studies? Of controlled trials identified in five separate subject areas within perinatal medicine, between 20 and 50 percent were identified using MEDLINE, compared with between 85 and 100 percent using the Oxford Database of Perinatal Trials. Chalmers et al. 1989

6. Retrieval bias What can be done about it? • Systematic and comprehensive search for eligible reports How can it be prevented? • Use of structured abstracts by investigators • Improvement, extension and further development of • Bibliographic retrieval systems • Explicit search criteria

7. Publication bias • A tendency among investigators, peer reviewers and journal editors to allow the direction and statistical significance of research findings to influence decisions regarding submission and acceptance for publication

8. Publication bias Examples Analysis of relative risks derived from studies examining the relationship between passive smoking and lung cancer suggests that studies in which no relationship has been detected may remain unpublished. Vandenbrouke, Br Med J 1988;296:391-392 However, estimates of the effect of these suggests that they would have little impact on the weight of evidence against passive smoking, 2004 A survey of authors or published reports of randomized trials revealed that between a quarter and a fifth of the trials that they have ever conducted had never been published, and that trials in which a new treatment had been found to be a superior to a standard treatment were more likely than others to have been published. Dickersin et al.Contr Clin Trials1987;8:343-353

9. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ  2003;326:1167-1170 (31 May),Joel Lexchin, Lisa A Bero, Benjamin Djulbegovic, Otavio Clark • Results 30 studies were included (1966 – 2002). Research funded by drug companies was less likely to be published than research funded by other sources.Studies sponsored by pharmaceutical companies were more likely to have outcomes favouring the sponsor than were studies with other sponsors (odds ratio 4.05; 95% confidence interval 2.98 to 5.51; 18 comparisons). None of the 13 studies that analysed methods reported that studies funded by industry was of poorer quality.

10. Publication bias • What can be done about it? • Grey literature • Contact authors

11. Pitfalls of reviews • “Current medical reviews do not routinely use scientific methods to identify, assess and synthesise information.” Mulrow, 1987 • 50 reviews in 4 major journals, 1985-86 • no statement of methods 49 • summary inappropriate 47

12. How can the situation improve? • Better reviews (high quality, more relevant) • Cochrane collaboration • Systematic reviews • Improved access to reviews • CCPC, Databases, Effectiveness Bulletins • Readers more skilled in making sense of reviews • CASP

13. What is a systematic review? A review in which evidence on a topic has been systematically identified and summarised according to predetermined criteria. • Specific clinical questions • Predefined explicit methodology • Reproducible • Usually review of RCTs

14. Meta-analysis - statistical principles The use of statistical methods to summarise the results of independent studies into a single estimate giving more weight to results from larger studies. • No direct comparison of patients • Summary statistics are calculated for each trial • Individual estimates are pooled - overall pooled estimate (if appropriate) • Gives more precise estimate of effect size • Interpretation / odds ratios

15. Types of review Reviews Systematic reviews Meta-analysis

16. Why use systematic reviews? • Volume of literature • Provide a basis for rational decision making • Are health care effects consistent? • Limit bias and reduce random error • Provides more reliable results • Required for ethics committees, funding agencies

17. Aim: To help people make well-informed decisions about health care by preparing, maintaining and promoting accessibility of systematic reviews of the effects of health care interventions. www.cochrane.org

18. 6 principles of Cochrane • Collaboration • 5000 people in 50 countries • Building on peoples’ existing enthusiasm and interests • 50 Cochrane Review Groups - focus on particular areas of health • Minimising duplication effort • Avoidance of bias • Keeping up to date • Ensuring access

19. Format for a (Cochrane) review • Contact Cochrane • Develop a protocol • Formulate the problem • Locate and select studies • Critical appraisal of studies • Score for selection, attrition, performance, detection, blinding biases • Collecting data • Analysing and presenting results • Interpreting results • Improving and updating reviews

20. What is a Cochrane Review?http://www.cochrane.org/reviews/revstruc.htm • Cochrane Reviews investigate the effects of interventions for prevention, treatment and rehabilitation in a healthcare setting. They are designed to facilitate the choices that doctors, patients, policy makers and others face in health care. Most Cochrane Reviews are based on randomized controlled trials, but other types of evidence may also be taken into account, if appropriate. Cochrane reviews have the following general features:

21. General features of a Cochrane review • Structured format • Detailed methods section • The quality of clinical studies to be incorporated into a review is carefully considered, using predefined criteria. • A thorough and systematic search strategy, which includes searches for unpublished and non-English records • If the data collected in a review are of sufficient quality and similar enough, they are summarised statistically in a meta-analysis, • generally provides a better overall estimate of a clinical effect than the results from individual studies. • Reviews aim to be relatively easy to understand for non-experts • Multinational editorial teams try to ensure that a review is applicable in different parts of the world • Reviews are updatable. • Results from newly completed or identified clinical trials can be incorporated into the review after publication. Additionally, readers can send in comments and criticisms to any review, and reviews may be changed accordingly to improve their quality.

22. Systematic review protocol • State objectives and eligibility criteria • Identify potentially eligible studies • Apply eligibility criteria • Refine protocol • Publish protocol

23. Elements of a systematic review • Define the clinical question • Identify all completed studies, un/published • Select the studies that meet scientific validity criteria • Look for evidence of bias • Describe the scientific quality of the studies • Assess if quality systematically related to results • Describe studies with a forest plot • Assess if similar enough to justify combining results • Calculate summary measure of effect & CI Fletcher & Fletcher T12.1

24. How can we appraise a review? • How can we systematically appraise a review? • What questions should we ask?

25. 10 questions to appraise a review Critical Appraisal Skills Programme (CASP) http://www.phru.org.uk/~casp/ • Three basic types of question... • Is it trustworthy? – Validity • Screening questions • Detailed questions on methodology • What does it say? – Results • Will it help? – Relevance

26. Is it trustworthy? – Validity • Screening questions • Did review address a clearly focused issue? • Did authors review right type of study? • Detailed questions on methodology • Were all important relevant studies included? • Did reviewers do enough to assess the quality of the included studies? • Randomisation system, scoring system, >1 assessor • If meta-analysis performed, was it reasonable to combine the results?

27. What does it say - Results • Rates • Relative and absolute risk, numbers needed to treat + confidence intervals • Clinical significance • Statistical significance

28. Will it help? – Relevance(What service providers should ask) • Is my patient... • Are the interventions available to my patient... • Are the outcomes relevant to my patient... .....sufficiently different from people in the review to allow me to consider the findings inapplicable?

29. Design literature search for:Quantitative systematic review of randomised controlled trials comparing antibiotic with placebo for acute cough in adults.(BMJ 1998;316:906-10) Fahey T. et al. • Aim: • To establish whether antibiotics are effective in the treatment of acute cough in the community

30. Literature search • Medical subject headings (MeSH) • Databases • Language? • Study type • Patients

31. Literature search • Medical subject headings (MeSH) • Cough, bronchitis, sputum, respiratory tract infections, chest infection • Databases • Medline, EMBASE, Science Citation Index, Cochrane Controlled Trials Register • Contacted authors - ?know of unpublished trials • UK drug companies ?unpublished trials • Language - not just English

32. Literature search Inclusion/exclusion criteria • Study type • Prospective trials, formal or quasi-randomisation • Placebo controlled • Patients • >12 years • Family practice clinic, community based o/px dept., hospital o/px dept. • Acute cough, with/out sputum, no antibiotic in preceding week • COAD excluded

34. Conclusions? • Treatment with antibiotic • Does not affect resolution of cough or alter course of illness • Benefits • Marginal for most patients with acute cough • May be outweighed by side effects (RR=1.51, CI 0.86 to 2.64)