EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis)

1. EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis) Dejana Braithwaite Assistant Professor UCSF Department of Epidemiology and Biostatistics April 11, 2013

2. Agenda • Introduction and definitions • Steps of a systematic review • Issues/Controversies • Conclusions

3. What’s a Systematic Review? “A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” Cochrane Collaboration

4. …and meta-analysis? Statistical combination of >= 2 studies to produce single estimate of effect of exposure

5. Meta-analyses IPD* Systematic reviews *IPD= individual participant data

6. The Cochrane CollaborationInternational systematic review initiative • Archie Cochrane’s vision led to the opening of the first Cochrane centre (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993 Source: http://www.cochrane.org/cochrane/archieco.htm

7. Meta-analysis news coverage Sattar N, et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet. 2010 Feb 27;375(9716):735-42.

9. Systematic review • Driven by evidence-based medicine movement and Cochrane collaboration • Advantages: • Reduces bias • Replicable • Resolves controversy between conflicting findings • Provides reliable basis for decision making

10. Use of meta-analysis as a prelude to clinical trials • Define pre-trial expected effect sizes sample size estimation • Determine effect estimates in key subgroups (e.g. based on gender, race/ethnicity or age) • Identify sources of heterogeneity in prior studies • Address these sources in design phase of new trial

11. Use of meta-analysis in study designs that are not clinical trials • Observational studies • (e.g. case control, cohorts, cross-sectional prevalence studies, etc.) • Studies evaluating diagnostic tests (sensitivity, specificity, predictive value) • “IPD” = individual patient data studies • Qualitative studies (meta-ethnography)

12. Resources required for systematic reviewing • Can be time consuming • Team science (to reduce bias) • Bibliographic software (e.g. Endnote) • Statistical software (if appropriate)

13. Citations Retrieved for a Meta-analysis and Total Hours Required to Complete the Meta-analysis* *IE Allen & I Olkin: JAMA. 1999;282(7):634-635. doi:10-1001/pubs.JAMA-ISSN-0098-7484-282-7-jbk0818 The mean total number of hours was 1139 (median, 1110), with a wide range from 216 to 2518 hours. • Pre-analysis search, retrieval, and database development: 588 (337) hours; • statistical analysis & validation: 144 (106) hours; • report and manuscript writing: 206 (125) hours; • other (administrative): 201 (193) hours. Total time=721 + 0.243x − 0.0000123x2, where x is the number of citations before exclusion criteria are applied. 

14. 5 steps of a systematic review 1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria 3. Assessing the quality of studies 4. Summarizing evidence Analysis 5. Interpreting the findings

15. Free-form question: Is it safe to provide population-wide drinking water fluoridation to prevent caries?Structured question: The populations—Populations receiving drinking water sourced through a public water supplyThe interventions or exposures—Fluoridation of drinking water (natural or artificial) compared with non-fluoridated waterThe outcomes—Cancer is the main outcome of interest for the debate in your health authorityThe study designs—Comparative studies of any design examining the harmful outcomes in at least two population groups, one with fluoridated drinking water and the other without.

16. Components of a research question (PICOT) • Outcome: • Risk or protective • Dichotomous or • continuous • Type: mortality, quality of life, etc. • Type of Study: • RCTs • Cohort • Case-control • Cross-sectional • All • Patient: • Disease or condition • Demographic characteristics • Intervention (or “Exposure”): • Type of intervention • Dose, duration, timing, etc. • Comparison: • Absence of risk or treatment • Placebo or alternative therapy

17. Formulation of an etiology question Risk factor/Exposure Outcome Is smoking a risk factor for breast cancer? Patient Exposure Are people who smoke regularly at a greater risk of developing breast cancer as compared to those who do not smoke? + cohort & case-control studies Outcome Comparison

18. Protocol • Background • Objectives • Pre-determined selection criteria • Planned search strategy • Planned data abstraction • Proposed method of synthesis of findings

19. 5 steps of a systematic review 1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria 3. Assessing the quality of studies 4. Summarizing evidence Analysis 5. Interpreting the findings

20. Where to locate studies • Subject specific: • PsychINFO • CINAHL At least two of these: • Pubmed • Web of Science • EMBASE • Cochrane Central

21. Additional sources to identify studies for systematic reviews Reference lists of retrieved articles Manual searching of relevant publications Experts in the field Corresponding or first authors of published studies identified for the systematic review

22. Issues to consider Publication bias Search bias

23. Inclusion/exclusion criteria P - Population I - Intervention C - Comparison (if necessary) O - Outcome T - Type of study (if necessary) Subject headings OR Textwords To find studies using all of the PICO elements: P and I and C and O (and T)

24. Exclusion criteria • Keep log of excluded studies • Note reasons for exclusion • Have eligibility checked by more than one reviewer • Develop strategy to resolve disagreements

25. PRISMA-based search strategy flow-chart Braithwaite et al Breast Ca Res Treatm 2012

26. EXAMPLE: SEARCH STRATEGY You are interested in evaluating the benefits and harms associated with the use of screening mammography in women aged 70 and older. You need to determine what literature is available to conduct your review.

27. Quiz 1: You decide to conduct a systematic review (SR). What are two ways in which SRs differ from narrative reviews? • They are the same thing • Systematic reviews use comprehensive searches and explicit methods • Systematic reviews always employ quantitative synthesis (meta-analysis)

28. Quiz 1 answers: SRs versus narrative reviews • They are the Same – Incorrect since narrative reviews do not employ explicit methods • Comprehensive Search and Explicit Methods – Correct! • Meta-analysis – Incorrect since SRs may or may not employ quantitative synthesis

29. Quiz 2: When conducting a comprehensive search, what do you need to assure? • You balanced precision and recall • You have employed every possible relevant citation regardless of the workload involved • That you limit the search so that only relevant citations are found

30. Quiz 2 answers: When conducting a comprehensive search, what do you need to assure? • Balancing precision and recall - This is correct! • Precision - proportion of retrieved articles that are relevant. • Recall - proportion of potentially relevant articles retrieved by the search. • You’ve Gotten it All– incorrect since this is unnecessarily onerous • Only Have Relevant Citations in your Search – incorrect since overly restrictive strategy may miss citations

31. Quiz 4: You try to decide whether to do one search or two searches (one for benefits and another for harms). Which of the following would cause you to perform two searches? • Performing two searches is redundant and should be avoided • You decide a priori to only allow RCT data when evaluating benefits but allowing RCTs and observational trials for harms • If the available literature base on the topic is extensive

32. Quiz 4 answers: one versus multiple searches • Never Do Multiple Searches - This is incorrect. Extensive literature may necessitate multiple searches. • Separate Benefits and Harms Search - This is correct! Benefits searches usually limited to RCTs due to greater internal validity. Harms searches usually broader. • When the available literature is extensive, two searches improves precision without appreciably impacting recall.

33. Quiz 5: You decide to use hand searching of references from identified SRs, studies, and abstract booklets from prominent meetings within the field of interest. Why would you do this? • Hand searching can capture citations that are not indexed or are improperly indexed. • Limiting hand searching to these sources can target this time intensive activity to areas where the yield will be the greatest.

34. Quiz 5: Value of hand searching • Both are Correct. Hand searching can yield up to 13-25% of citations that do not come up in database searches. • Since hand searching is the most time intensive search strategy, it should be targeted to maximize the yield.

35. Quiz 6: You decide to search the FDA website and ClinicalTrials.gov to identify studies that might be appropriate for your systematic review. These website are an example of: • Grey literature • Informally published written material

36. Quiz 6 answers: Grey Literature • Grey Literature or informally published written material – both are correct! • This can be important for identifying studies that were conducted but are, as of now, unpublished or incompletely published. • Can minimize publication bias.

37. Search strategy - summary • A comprehensive understanding of the topic, the use of multiple searches, multiple databases, use of hand searching and grey literature, and development of a transparent reporting structure will give readers confidence in your systematic review.

38. 5 steps of a systematic review 1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria 3. Assessing the quality of studies 4. Summarizing evidence Analysis 5. Interpreting the findings

39. Principles of quality assessment • Quantitative studies • Internal Validity • Allocation bias confounding • Attrition • Statistical analysis • Intervention integrity • Withdrawals and dropouts • External Validity • generalizability or applicability

40. Quality Scoring of Publications • Examines design and reporting • Jadad scoring technique (5 point Max) • Randomization (1 + 1 points) • Blinding (1 + 1 points) • Withdrawals (1 point) (1996, Controlled Clinical Trials) This followed the earlier 3 page quality scoring technique suggested by: Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A: A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2:31-49, 1981 Slide courtesy of I. Elaine Allen, PhD http://en.wikipedia.org/wiki/Jadad_scale

41. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement • Evidence-based minimum set of items for reporting in SRs and meta-analyses • The aim of the PRISMA Statement is to help authors improve the reporting of SRs and meta-analyses • Focused on randomized trials but can also be used as a basis for reporting SRs of other types of research www.prisma-statement.org/

42. Recruit participants SELECTION BIAS ALLOCATION BIAS Allocate to intervention and control groups Intervention group Control group CONFOUNDING INTEGRITY OF INTERVENTION Implement intervention Implement intervention INTENTION-TO-TREAT Follow-up participants Follow-up participants WITHDRAWALS/ DROP OUTS Measure outcomes BLINDING OUTCOME ASSESSORS Measure outcomes DATA COLLECTION METHODS Analyze outcomes Analyze outcomes STATISTICAL ANALYSIS

43. 5 steps of a systematic review 1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria 3. Assessing the quality of studies 4. Summarizing evidence Analysis 5. Interpreting the findings

44. Design and pilot data abstraction form • Consider >1 reviewer • Consider blinding of observers to authors, institutions and journals Data abstraction

45. publication details study design population details (n, characteristics) intervention details setting outcomes and findings Data abstraction elements

46. Synthesizing the Evidence • NARRATIVE SYNTHESIS • findings summarized and explained qualitatively • META-ANALYSIS • findings summarized and then combined statistically

47. Is there heterogeneity? No Yes Meta-analysis Narrative synthesis Deal with heterogeneity? (e.g. subgroup analysis)

50. 5 steps of a systematic review 1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria 3. Assessing the quality of studies 4. Summarizing evidence Analysis 5. Interpreting the findings