Systematic Reviews and Meta-Analyses

1. Systematic Reviews and Meta-Analyses And the Cochrane Library Ritz Kakuma, MSc (PhD Candidate)Department of Epidemiology & Biostatistics McGill University

2. Problems with Today's Medical Literature • Most studies are too small • Inconclusive, often conflicting results • Traditional reviews are unstructured & subjective • Biased conclusions

3. WHO CAN KEEP UP? • Over 345,000 trials published to date • Over 20,000 new trials published annually Help!

4. For General Physiciansto keep current: Read 19 new articles per day which appear in medical journals 19 x 2 hrs (Critical Appraisal) = 38 hrs per day Davidoff F et al. (1995) EBM; A new journal to help doctors identify the information they need. BMJ 310:1085-86.

5. 1996 G. WELCH, S. GABBEReview of Statistics Usage in the Amer J Obstet Gynecol 175;1138-41 Statistical Quality of Medical Research gets low marks! • All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6) • 31.7% (46/145) had inappropriate statistics • 27 of 46 papers had serious flaws RESULTS???

6. Problems with Standard Reviews • Lack of scientific purpose (question) • Undocumented methods of literature search • Unstated criteria for selecting studies • No methodological assessment of selected studies • Inadequate assessment of inter-study differences in results • No attempt at quantitative synthesis (pooling) to take advantage of increased power

7. Why Systematic Reviews? • Help to deal with the volume of literature • Help resolve conflicting results • Scientific rather than subjective summarization of literature • Can reveal new evidence • Identify knowledge gaps • More reliable evidence with which to aid decision making • Guide clinical research by providing new hypotheses

8. The Cochrane Collaboration - origins • Archie Cochrane • “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.” • Pregnancy and childbirth work - 1980s

9. Canadian Cochrane Network & Centre Aims and objectives of the CC “The Cochrane Collaboration is an international organization that aims to help people make well­informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions” • Founded 1993

10. Canadian Cochrane Network & Centre CC built on 10 Principles • collaboration • building on the enthusiasm of individuals • avoiding duplication • minimizing bias • keeping up to date • striving for relevance • promoting access • ensuring quality • continuity • enabling wide participation

11. Canadian Cochrane Network & Centre Organization of the CC • Cochrane Centres (n=12) • Collaborative Review groups (n=51) • Fields (n=9) • Networks (n=1) • Methods working groups (n=10)

12. Canadian Cochrane Network & Centre Cochrane Centres • Australasian CC • Brazilian CC • Canadian CC • Chinese CC • Dutch CC • German CC • Iberoamerican CC • Italian CC • Nordic CC • South African CC • UK CC • US CC (Rhode Island, Boston, San Francisco Branches)

13. Canadian Cochrane Network & Centre Cochrane Review Groups Infectious Diseases Group Inflammatory Bowel Disease Group Injuries Group Lung Cancer Group Menstrual Disorders and Subfertility Group Metabolic and Endocrine Disorders Group Methodology Review Group Movement Disorders Group Multiple Sclerosis Group Musculoskeletal Group Musculoskeletal Injuries Group Neonatal Group Neuromuscular Disease Group Oral Health Group Pain, Palliative and Supportive Care Peripheral Vascular Diseases Group Pregnancy and Childbirth Group Prostatic Diseases and Urologic Cancers Group Renal Group Schizophrenia Group Sexually Transmitted Diseases Group Skin Group Stroke Group Subfertility Group (see Menstrual Disorders) Tobacco Addiction Group Upper Gastrointestinal & Pancreatic Diseases Grp Wounds Group Acute Respiratory Infections Group Airways Group Anaesthesia Group Back Group Breast Cancer Group Colorectal Cancer Group Consumers and Communication Group Cystic Fibrosis and Genetic Disorders Group Dementia and Cognitive Improvement Group Depression, Anxiety and Neurosis Group Developmental, Psychosocial & Learning Problems Group Drugs and Alcohol Group Ear, Nose and Throat Disorders Group Effective Practice and Organisation of Care Group Epilepsy Group Eyes and Vision Group Fertility Regulation Group Gynaecological Cancer Group Haematological Malignancies Group Heart Group Hepato-Biliary Group HIV/AIDS Group Hypertension Group Incontinence Group

14. Canadian Cochrane Network & Centre Cochrane Fields & Network • Cancer Network • Child Health • Complementary Medicine • Health Care of Older People • Health Promotion and Public Health • Neurological Network • Primary Health Care • Rehabilitation and Related Therapies • Vaccines Field • Consumer Network

15. Canadian Cochrane Network & Centre Cochrane Methods Groups • Applicability and Recommendations • Health Economics • Health-Related Quality of Life • Individual Patient Data Meta-Analyses • Non-randomised Studies • Prospective Meta-Analysis • Qualitative Methods • Reporting Bias Methods • Screening and Diagnostic Tests • Statistical Methods

16. Canadian Cochrane Network & Centre Cochrane Activities • Produce and update systematic reviews • Hand search for RCTs • Medline (and others) enhancement • Review methodology • COCHRANE LIBRARY

17. Canadian Cochrane Network & Centre When should you use the CLIB? For questions on effectiveness • What is the effectiveness of treatment x • What is an effective treatment for y • Is z effective in treating y • Is z better than x at treating y

18. Canadian Cochrane Network & Centre When not to use the Cochrane Library General healthcare questions • causal, prognosis, epidemiology, etc. • Statistics (prevalence and incidence) • Primary research other than RCTs • Guidelines • Current research

19. Canadian Cochrane Network & Centre The Cochrane Database of Systematic Reviews (CDSR) Complete Reviews • full text, regularly updated systematic reviews of the effects of health care (1,837 reviews) • prepared and maintained by the Collaboration Review Groups Protocols • protocols of reviews currently being prepared, incl. expected date of completion (1,344 protocols) • includes background, objectives and methods sections

20. Canadian Cochrane Network & Centre The Database of Abstracts of Reviews of Effectiveness (DARE) prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK. Abstracts of quality assessed systematic reviews • structured abstracts assessing the quality of previously published SRs & summarizing findings (3,484 reviews) Other reviews: bibliographic details only • references to published SRs NOT assessed for quality (800 reviews)

21. Canadian Cochrane Network & Centre The Cochrane Central Register of Controlled Trials (CENTRAL) References • Reference list of ALL identified published randomized trials • Latest issue 378,160 RCTs (Medline: 81,705 articles identified as RCT publication type)

22. Canadian Cochrane Network & Centre The Cochrane Database of Methodology Reviews (CDMR) Complete Reviews • Full-text SRs of methodological studies (9 reviews) • Highly structured and systematic, covering a specific and well-defined area of methodology • prepared and maintained by the Cochrane Methodology Review Groups Protocol • protocols of reviews currently being prepared, incl. expected date of completion (7 protocols) • includes background, objectives and methods sections

23. Canadian Cochrane Network & Centre Cochrane Review Methodology Database (CRMD) References • published reports of empirical studies of methods used in reviews (4,626 reports) • methodological studies directly relevant to conducting a review

24. Canadian Cochrane Network & Centre Searching the CLIB Some basics… ALL contents of ALL records in ALL databases are searched Ignores: punctuation & numbers Boolean terms: AND, OR, NEXT, NEAR – within 6 words both ways, NOT Restricting searches • ‘Options’ page and choose desired restrictions • At the end of the term, add: :AU – Author :TI – Title :ME - MeSH terms :AB – Abstract :KY - Keywords

25. Canadian Cochrane Network & Centre Searching the CLIB MeSH • Keywords drawn from MeSH thesaurus of U.S. NLM • Accompanies some but not all records • Organized hierarchically in ‘trees’ • Permuted Index – an index of all words that appear in the MeSH thesaurus  used to located specific MeSH terms

26. Canadian Cochrane Network & Centre At McGill University and affiliated sites: The pathway to the CLIB is: McGill HSL library homepage > Databases > > Cochrane Library > web (no login or password nec.) http://www.cochranelibrary.com/enter/

38. Canadian Cochrane Network & Centre Useful Materials • The Cochrane Library: Self Training Guide • Interpretation of Odd-ratio diagrams • WWW links, HTA database • Cochrane Handbook • User manual • Glossary, etc... DOWNLOAD http://www.cochrane.org/resources/training.htm

39. Quality of Cochrane Reviews • Comparison of 36 Cochrane reviews with 39 paper-based journals  Cochrane reviews less prone to bias (Jadad et al. 1998) • Explicit reporting of eligibility criteria (35/36 vs 18/39) • Assessed trial quality (36/36 vs 7/39) • No language restriction (36/36 vs 32/39)

40. Quality of Cochrane Reviews • Also not perfect (Olsen et al. 2001) • 52 Cochrane reviews from 1998 • 18% - conclusions not backed up by evidence • All overestimated effect of intervention

41. Discordant Reviews • Direction of effect • Significance • Magnitude of effect • Interpretation of results  Numerous possible reasons

42. Sources of discordance • Different research question • Target population • Intervention being studied • Outcome measures • Setting • Study selection • Search strategy • Eligibility criteria

43. Sources of discordance • Data extraction • Methods of measuring outcomes and endpoints • Extent of human error • Quality assessment method • 25 scales and 9 checklists avail for assessing RCT quality (Moher et al. 1996) • Inconsistent quality depending on instrument used • QUOROM (Quality of Reporting of Meta-Analyses) = most comprehensive(Moher et al. 1999, Shea et al. 2010)

44. Sources of discordance • Analysis • Appropriateness of combining results • Method of synthesis: Descriptive, meta-analysis • Statistical methods • Bayesian • Meta-regression • Frequentist • Interpretation of evidence

45. Decision Algorithm for interpreting discordant reviews

46. Screening for breast cancer with mammography among women aged 50-69 yearsReport prepared for the Breast Cancer Screening Unit,Cancer Branch, Health CanadaRitsuko KakumaMarch 2002

47. MAs on mammography screening for breast cancer: • Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656. • Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154. • Olsen O & Gøtzsche PC. Is screening for breast cancer with mammography justifiable? Lancet 2000.

48. Fletcher et al. review Research Question: • To assess current state of knowledge about BRCA screening, identify knowledge gaps Study Sample: • Women aged 40-79 years stratified into 3 groups: 40-49, 50-69, 70+

49. Fletcher et al. review Methodological issues: • Int’l Workshop on screening for BRCA in Feb 1993 (by NCI) • W/S participant characteristics not reported • Study selection method not described • Data source: published and unpublished data provided by the w/s participants • Validity of trials assessed: • Randomization, Compliance, Contamination • Quality and frequency of screening test • Adequacy of FU • Validity of outcome assessment • Generalizability of results

50. Fletcher et al. review Results/Conclusions: • 7 trials identified • For age 50-69 age group, only 5 trials provided data • New York, Two-County, Malmo, Edinburgh and Canada • No statistical pooling of data • Together, Fletcher et al. concluded that there was substantial benefit of screening on BRCA mortality  reduction of 30% after 10-12 years of FU