Systematic Reviews and Meta-Analyses

1. Systematic Reviews and Meta-Analyses And the Cochrane Library Ritz Kakuma, MSc (PhD Candidate)Department of Epidemiology & Biostatistics McGill University

2. Outline • Background on SR/MA • Cochrane Collaboration • Cochrane Library • Example on Mammography Screening for Breast Cancer

3. Problems with Today's Medical Literature • Most studies are too small • Inconclusive, often conflicting results • Traditional reviews are unstructured & subjective • Biased conclusions

4. WHO CAN KEEP UP? • Over 425,000 trials published to date • Over 20,000 new trials published annually Help!

5. For General Physiciansto keep current: Read 19 new articles per day which appear in medical journals 19 x 2 hrs (Critical Appraisal) = 38 hrs per day Davidoff F et al. (1995) EBM; A new journal to help doctors identify the information they need. BMJ 310:1085-86.

6. 2002 G. WELCH, S. GABBEStatistics Usage in the Amer J Obstet Gynecol: has anything changed? 180;584-6 Statistical Quality of Medical Research getting better… • All clinical papers: Jan - June 1994 (Vol. 170, No. 1 to 6) vs. Jan – June 1999 (Vol. 180, No. 1 to 6) • Inappropriate statistics used in 31.7% (46/145) in 1994 and 9.8% (19/195) in 1999. RESULTS???

7. Problems with Standard Reviews • Lack of scientific purpose (question) • Undocumented methods of literature search • Unstated criteria for selecting studies • No methodological assessment of selected studies • Inadequate assessment of inter-study differences in results • No attempt at quantitative synthesis (pooling) to take advantage of increased power

8. Why Systematic Reviews? • Help to deal with the volume of literature • Help resolve conflicting results • Scientific rather than subjective summarization of literature • Can reveal new evidence • Identify knowledge gaps • More reliable evidence with which to aid decision making • Guide clinical research by providing new hypotheses

9. The Cochrane Collaboration - origins • Archie Cochrane • “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.” • Pregnancy and childbirth work - 1980s • Founded 1993

10. Canadian Cochrane Network & Centre Aims and objectives of the CC “The Cochrane Collaboration is an international organization that aims to help people make well­informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions”

11. Canadian Cochrane Network & Centre CC built on 10 Principles • collaboration • building on the enthusiasm of individuals • avoiding duplication • minimizing bias • keeping up to date • striving for relevance • promoting access • ensuring quality • continuity • enabling wide participation

12. Canadian Cochrane Network & Centre Organization of the CC • Cochrane Centres (n=12) • Collaborative Review groups (n=51) • Fields (n=11) • Networks (n=1) • Methods working groups (n=10)

13. Canadian Cochrane Network & Centre Cochrane Centres • Australasian CC • Brazilian CC • Canadian CC • Chinese CC • Dutch CC • German CC • Iberoamerican CC • Italian CC • Nordic CC • South African CC • UK CC • US CC (Rhode Island, Boston, San Francisco Branches)

14. Canadian Cochrane Network & Centre Cochrane Review Groups Infectious Diseases Group Inflammatory Bowel Disease Group Injuries Group Lung Cancer Group Menstrual Disorders and Subfertility Group Metabolic and Endocrine Disorders Group Methodology Review Group Movement Disorders Group Multiple Sclerosis Group Musculoskeletal Group Musculoskeletal Injuries Group Neonatal Group Neuromuscular Disease Group Oral Health Group Pain, Palliative and Supportive Care Peripheral Vascular Diseases Group Pregnancy and Childbirth Group Prostatic Diseases and Urologic Cancers Group Renal Group Schizophrenia Group Sexually Transmitted Diseases Group Skin Group Stroke Group Subfertility Group (see Menstrual Disorders) Tobacco Addiction Group Upper Gastrointestinal & Pancreatic Diseases Grp Wounds Group Acute Respiratory Infections Group Airways Group Anaesthesia Group Back Group Breast Cancer Group Colorectal Cancer Group Consumers and Communication Group Cystic Fibrosis and Genetic Disorders Group Dementia and Cognitive Improvement Group Depression, Anxiety and Neurosis Group Developmental, Psychosocial & Learning Problems Group Drugs and Alcohol Group Ear, Nose and Throat Disorders Group Effective Practice and Organisation of Care Group Epilepsy Group Eyes and Vision Group Fertility Regulation Group Gynaecological Cancer Group Haematological Malignancies Group Heart Group Hepato-Biliary Group HIV/AIDS Group Hypertension Group Incontinence Group

15. Canadian Cochrane Network & Centre Cochrane Fields & Network • Cancer Network Field • Child Health Field • Complementary Medicine Field • Health Care of Older People Field • Health Promotion and Public Health Field • Neurological Network Field • Occupational Health Field • Prehospital & Emergency Health Field • Primary Health Care Field • Rehabilitation and Related Therapies Field • Vaccines Field • Consumer Network

16. Canadian Cochrane Network & Centre Cochrane Methods Groups • Applicability and Recommendations • Health Economics • Health-Related Quality of Life • Individual Patient Data Meta-Analyses • Non-randomised Studies • Prospective Meta-Analysis • Qualitative Methods • Reporting Bias Methods • Screening and Diagnostic Tests • Statistical Methods

17. Canadian Cochrane Network & Centre Cochrane Activities • Produce and update systematic reviews • Hand search for RCTs • Medline (and others) enhancement • Review methodology • COCHRANE LIBRARY

18. Canadian Cochrane Network & Centre The Cochrane Library • unique source of reliable and up-to-date information on the effects of interventions in health care • To provide information and evidence to support decisions taken in health care and to inform those receiving care • Published on a quarterly basis

19. Canadian Cochrane Network & Centre When should you use the CLIB? For questions on effectiveness • What is the effectiveness of treatment x • What is an effective treatment for y • Is z effective in treating y • Is z better than x at treating y

20. Canadian Cochrane Network & Centre When not to use the Cochrane Library General healthcare questions • causal, prognosis, epidemiology, etc. • Statistics (prevalence and incidence) • Primary research other than RCTs • Guidelines • Current research

21. Canadian Cochrane Network & Centre The Cochrane Database of Systematic Reviews (CDSR) Complete Reviews • full text, regularly updated systematic reviews of the effects of health care (2,170 reviews) • prepared and maintained by the Collaboration Review Groups Protocols • protocols of reviews currently being prepared, incl. expected date of completion (1,500 protocols) • includes background, objectives and methods sections

22. Canadian Cochrane Network & Centre The Database of Abstracts of Reviews of Effectiveness (DARE) prepared by the National Health Services Centre for Reviews and Dissemination at the University of York, UK. Abstracts of quality assessed systematic reviews • structured abstracts assessing the quality of previously published SRs & summarizing findings (4,118 reviews) Other reviews: bibliographic details only • references to published SRs NOT assessed for quality (800 reviews)

23. Canadian Cochrane Network & Centre The Cochrane Central Register of Controlled Trials (CENTRAL) References • Reference list of ALL identified published randomized trials • Latest issue 427,807 RCTs (Medline: 97,827 articles identified as RCT publication type)

24. Canadian Cochrane Network & Centre The Cochrane Database of Methodology Reviews (CDMR) Complete Reviews • Full-text SRs of methodological studies (10 reviews) • Highly structured and systematic, covering a specific and well-defined area of methodology • prepared and maintained by the Cochrane Methodology Review Groups Protocol • protocols of reviews currently being prepared, incl. expected date of completion (8 protocols) • includes background, objectives and methods sections

25. Canadian Cochrane Network & Centre Cochrane Methodology Register (CMR) References • published reports of empirical studies of methods used in reviews (5,968 reports) • methodological studies directly relevant to conducting a review

26. Canadian Cochrane Network & Centre Health Technology Assessment Database (HTA) References • HTA covers prevention and rehabilitation, vaccines, pharmaceuticals and devices, medical and surgical procedures and the systems within which health is protected and maintained • Ongoing projects and completed publications from HTA organizations (4,395 citations)

27. Canadian Cochrane Network & Centre NHS Economic Evaluation Database (NHS EED) References • Structured abstracts of articles reporting economic evaluations of health care interventions • Bibliographic details of articles on relevant topics (i.e., burden of illness, economic methodology, reviews of economic evaluations) • N=15,041 citations

28. Canadian Cochrane Network & Centre Searching the CLIB Some basics… ALL contents of ALL records in ALL databases are searched Ignores: punctuation & numbers Boolean terms: AND, OR, NEXT, NEAR – within 6 words both ways, NOT Restricting searches • ‘Options’ page and choose desired restrictions • At the end of the term, add: :AU – Author :TI – Title :ME - MeSH terms :AB – Abstract :KY - Keywords

29. Canadian Cochrane Network & Centre Searching the CLIB MeSH • Keywords drawn from MeSH thesaurus of U.S. NLM • Accompanies some but not all records • Organized hierarchically in ‘trees’ • Permuted Index – an index of all words that appear in the MeSH thesaurus  used to located specific MeSH terms

30. Canadian Cochrane Network & Centre At McGill University and affiliated sites: The pathway to the CLIB is: McGill HSL library homepage > Databases > > Cochrane Library > web (no login or password nec.)

41. Canadian Cochrane Network & Centre Useful Materials • The Cochrane Library: Self Training Guide • Interpretation of Odd-ratio diagrams • WWW links, HTA database • Cochrane Handbook • User manual • Glossary, etc... DOWNLOAD http://www.cochrane.org/resources/training.htm

42. Quality of Cochrane Reviews • Comparison of 36 Cochrane reviews with 39 paper-based journals  Cochrane reviews less prone to bias (Jadad et al. 1998) • Explicit reporting of eligibility criteria (35/36 vs 18/39) • Assessed trial quality (36/36 vs 7/39) • No language restriction (36/36 vs 32/39)

43. Quality of Cochrane Reviews • Also not perfect (Olsen et al. 2001) • 52 Cochrane reviews from 1998 • 18% - conclusions not backed up by evidence • All overestimated effect of intervention

44. Discordant Reviews • Direction of effect • Significance • Magnitude of effect • Interpretation of results  Numerous possible reasons

45. Sources of discordance • Different research question • Target population • Intervention being studied • Outcome measures • Setting • Study selection • Search strategy • Eligibility criteria

46. Sources of discordance • Data extraction • Methods of measuring outcomes and endpoints • Extent of human error • Quality assessment method • 25 scales and 9 checklists avail for assessing RCT quality (Moher et al. 1996) • Inconsistent quality depending on instrument used • Additional 8 instruments (Juni et al. 1999) • QUOROM (Quality of Reporting of Meta-Analyses) = most comprehensive(Moher et al. 1999, Shea et al. 2001)

47. Sources of discordance • Analysis • Appropriateness of combining results • Method of synthesis: Descriptive, meta-analysis • Statistical methods • Bayesian • Meta-regression • Frequentist • Interpretation of evidence

48. Decision Algorithm for interpreting discordant reviews Jadad, Cook & Browman. A guide to interpreting discordant systematic reviews. CMAJ. 1997 May 15;156(10):1411-6.

49. Screening for breast cancer with mammography among women aged 50-69 yearsReport prepared for the Breast Cancer Screening Unit,Cancer Branch, Health CanadaRitsuko KakumaMarch 2002

50. SRs on mammography screening for breast cancer • Fletcher et al. Report of the International Workshop on Screening for Breast Cancer. J Natl Cancer Inst 1993; 85:1644-1656. • Kerlikowske et al. Efficacy of screening mammography. A meta-analysis. JAMA 1995; 273:149-154. • Olsen O & Gøtzsche PC. Is screening for breast cancer with mammography justifiable? Lancet 2000.