Conducting and Interpreting Systematic Reviews and Meta-Analyses

1. Conducting and Interpreting Systematic Reviews and Meta-Analyses July 12, 2007

2. Initial Notes about Reviews • Reviews are considered the “highest” form of evidence in the EBM evidence pyramid • Techniques/issues discussed here are different from the type of review (e.g., Psychological Bulletin) you are used to • Reviews are “research of research” (this is the derivation of the ‘meta’ component in ‘meta-analysis’) • As such, reviews must follow, and must make explicit, their methodology just like primary studies do

3. Evidence Pyramid for Treatment Effectiveness Questions Goal: Use best available evidence

4. Types of Reviews • Systematic Review: literature review focused on a single question which tries to identify, appraise, select and synthesize all high quality research evidence relevant to that question • Meta-analysis: a statistical technique often used in systematic reviews that enables the results from a number of studies to be combined into a common metric to determine the average effect of a given technique. Comparisons can then be made about the relative effectiveness of various techniques • Integrative Review: literature review that attempts to combine experimental and nonexperimental studies, or from experimental and theoretical work. Can be used to review theories, evidence, or to analyze methodological issues • Qualitative Review: methods for combining qualitative research studies. Techniques include meta-synthesis, formal grounded theory, and meta-ethnography

5. Review Stages • Problem formulation • Literature search • Data evaluation/extraction • Data analysis • Presentation

6. Systematic Reviews • Definition: summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies • Most evaluate the effectiveness of therapy, but systematic reviews of diagnostic test are also important • Systematic reviews are: • Systematic in identification of literature • Explicit in statement of objectives, materials, and methods • Reproducible in its methodology and conclusions • Best individual sources: Cochrane, PubMed Clinical Queries

7. Systematic Reviews: Search Strategies • Search Strategies • Electronic Databases (Medline, EMBASE, Cochrane; Clinical Trials Registry Database)http://ssrc.tums.ac.ir/SystematicReview/CTRDB.asp • Hand searching for published studies not in electronic databases • Checking reference lists • Contacting authors • Balance between comprehensiveness and precision • Document search strategy in final review • Additional resources: • Cochrane handbook: http://www.cochrane.org/resources/handbook/index.htm • Searchable database of research papers (dissertations and theses) http://dissertationsandtheses.com/ • Current Contents Connect http://scientific.thomson.com/products/ccc/

8. Systematic Reviews: Critical Appraisal • Critical appraisal = quality assessment • Is the evidence valid and relevant for your clinical question or situation? • Ways validity assessments are used • As a threshold for inclusion • As an explanation for differences in results between studies • As a sensitivity measure • As a means for weighting studies in statistical analyses • Dimensions of critical appraisal • Internal validity • Applicability • Precision (size of CI) • Useful resources • http://ssrc.tums.ac.ir/SystematicReview/Appraisal-Tools.asp • http://ssrc.tums.ac.ir/SystematicReview/RCTtools.asp

9. Systematic Reviews: Conclusions • Methodological Issues • Number of studies and participants • Size of treatment effect • Precision of treatment effect • Consistency of outcomes and results • Apparent dose-response relationships, if any • Generalizability and applicability of findings to other populations and settings

10. Meta-Analysis • Meta analysis is a set of statistical techniques for combining information from different studies to derive an overall estimate of a treatment's effect. • Potential issues (a “Catch 22”): • Publication bias • Varying quality of studies • Suggestions for high quality meta-analyses: • A formal protocol should be written specifying the exact question under investigation and describing the studies that will be included in the analysis. • All research, not just published research, should be included. The issue of “fugitive literature” is important. • Registries should be established so that studies can be tracked from their inception and not just on publication. This idea has been given a push so that drug companies would not be able to publish trials showing benefit from their products while suppressing those that do not. • Measures of study quality should be included • Clear eligibility criteria should be established • Many meta analytic techniques should be used and all results should be reported. A result would be considered reliable only if all of the techniques give the same result.

11. Steps in Statistical Analysis • Select measure of effect (rate difference, odds ratio, standardized mean differenceetc.) • RCT: rate difference (EER v. CER) or rate ratio • Case Control: odds or rate ratios • Standardized mean difference between E and C • Specify effects model: • Fixed: results generalizable to reviewed studies – assumes that there is an underlying effect running through all studies • Random: results generalizable to universe of such studies (computationally more intense) – assumes that the true effect estimate varies across studies

12. Meta-Analysis: Data Analytic Techniques • Dichotomous outcomes (e.g., cured v. not) • Mantel-Haenszel: pooled odds ratio across a strata of 2x2 tables • Analysis example: http://www.statsdirect.com/help/chi_square_tests/mh.htm

13. Meta-Analysis: Data Analytic Techniques (cont’d) • Continuous Data: Standardized difference as an example (e.g., Cohen’s d) • d = M1-M2/SD • Pooled SD of experimental and control groups is usually used; square root of the average of the two squared SD’s • Can be computed from t-test (d = 2t/df • Other measures: Hedges’s g

17. Therapeutic trials of streptokinase for MI

19. Van Emmerik, A.A.P., Kamphuis, J.A., Hulsbosch, A.M., & Emmelkamp, P.M.G. (2002). Single-session debriefing after psychological trauma: A meta-analysis. Lancet, 360, 766-771.

20. Integrative Reviews • Provide review of available literature according to clear methodological approach • Potential sources of bias • Incomplete literature search stage • Errors in data extraction • Inappropriate data analysis, particularly in combining experimental and nonexperimental methods

21. Integrative Reviews • Defining the review question clearly • Literature search issues • Obtaining needed literature often challenging • Publication bias • Identify maximum number of primary sources through multiple strategies; make strategies explicit (i.e., specify search terms) in review • Data evaluation issues • Explicit assessment of study quality – complex when included studies have vastly different designs • How to define quality of nonempirical studies?

22. Integrative Reviews (cont’d) • Data analysis issues • Developing a classification system for managing data from diverse methodologies • Classifying studies based on some logical, predetermined system (e.g., type of evidence, demographic characteristics, predetermined conceptual classification) and analyzing each subgroup sequentially

23. Qualitative Reviews • Qualitative research is typically nonexperimental, but it may be useful to include in systematic reviews because it evaluates important aspects of clinical research, often from the patient perspective (e.g., “views” research) • Relies on inductive rather than deductive reasoning • Qualitative research involves different measures of report quality than do RCT’s • Integrating quantitative and qualitative data sources is a key challenge to systematic reviewers

24. Types of Qualitative Data http://hsc.uwe.ac.uk/dataanalysis/qualVisDataEx.asp

25. Major Types of Qualitative Research • Participant Observation: used to assess the impact of healthcare events from the patient perspective • Interviews: used to assess range of ideas and concepts among individuals involved in healthcare delivery and receipt of healthcare services. Oral histories are one example • Interview example: http://hsc.uwe.ac.uk/dataanalysis/qualTextDataSelf.asp • Record review: used to assess phenomenology that leads to health care decisions (e.g., what factors lead to discharge or prescription of particular medications or behavioral intervention strategies) • Focus groups • Case studies

26. Qualitative Research: Analytic Strategy Examples • Constant Comparison Method: All data sources are compared to all others to search for commonalities or differences • “Grounded Theory” (Glaser, Strauss) – systematic method of generating theory from data • Ethnographic methods: How individuals or cultural groups conceptualize or enact health/illness or health-seeking

27. Integrating Qualitative and Quantitative Date in Systematic Reviews Thomas, et al., (2004). Integrating qualitative research with trials in systematic reviews. BMJ, 328, 1010-1012.