Systematic Reviews: Methods and Procedures

1. Systematic Reviews: Methods and Procedures George A. Wells Editor, Cochrane Musculoskeletal Review Group Department of Epidemiology and Community Medicine University of Ottawa Ottawa, Ontario, Canada

2. Meta-analysis: • Meta-analysis is a statistical analysis of a collection of studies • Meta-analysis methods focus on contrasting and comparing results from different studies in anticipation of identifying consistent patterns and sources of disagreements among these results • Primary objective: • Synthetic goal (estimation of summary effect) vs • Analytic goal (estimation of differences)

3. Systematic Review: • the application of scientific strategies that limit bias to the systematic assembly, critical appraisal and synthesis of all relevant studies on a specific topic • Meta-Analysis: • a systematic review that employs statistical methods to combine and summarize the results of several studies

4. Features of narrative reviews and systematic reviews NARRATIVESYSTEMATIC QUESTIONBroad Focused SOURCES/ Usually unspecified Comprehensive; SEARCH Possibly biased explicit SELECTION Unspecified; biased?Criterion-based; uniformly applied APPRAISAL Variable Rigourous SYNTHESIS Usually qualitative Quantitative INFERENCE Sometimes Usually evidence- evidence-based based

5. Steps of a Cochrane Systematic Review • Clearly formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report

6. What is the study objective • to validate results in a large population • to guide new studies • Pose question in both biologic and health care terms specifying with operational definitions • population • intervention • outcomes (both beneficial and harmful)

7. Inclusion Criteria • Study design • Population • Interventions • Outcomes

8. Steps of a Cochrane Systematic Review • Clearly formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report

9. Need a well formulated and co-ordinated effort • Seek guidance from a librarian • Specify language constraints • Requirements for comprehensiveness of search depends on the field and question to be addressed • Possible sources include: • computerized bibliographic database • review articles • abstracts • conference proceedings • dissertations • books • experts • granting agencies • trial registries • industry • journal handsearching

10. Procedure: • usually begin with searches of biblographic reports (citation indexes, abstract databases) • publications retrieved and references therein searched for more references • as a step to elimination of publication bias need information from unpublished research • databases of unpublished reports • clinical research registries • clinical trial registries • unpublished theses • conference indexes Published Reports (publication bias ie. tendency to publish statistically significant results)

11. Steps of a Cochrane Systematic Review • Clearly formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report

12. Study Selection • 2 independent reviewers select studies • Selection of studies addressing the question posed based on a priori specification of the population, intervention, outcomes and study design • Level of agreement: kappa • Differences resolved by consensus • Specify reasons for rejecting studies

13. Data Extraction • 2 independent reviewers extract data using predetermined forms • Patient characteristics • Study design and methods • Study results • Methodologic quality • Level of agreement: kappa • Differences resolved by consensus

14. Data Extraction …. • Be explicit, unbiased and reproducible • Include all relevant measures of benefit and harm of the intervention • Contact investigators of the studies for clarification in published methods etc. • Extract individual patient data when published data do not answer questions about: intention to treat analyses, time-to-event analyses, subgroups, dose-response relationships

15. Steps of a Cochrane Systematic Review • Well formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report

16. Description of Studies • Size of study • Characteristics of study patients • Details of specific interventions used • Details of outcomes assessed

17. Methodologic Quality Assessment • Can use as: • threshold for inclusion • possible explanation form heterogeneity • Base quality assessments on extent to which bias is minimized • Make quality assessment scoring systems transparent and parsimonious • Evaluate reproducibility of quality assessment • Report quality scoring system used

18. Study Random Blinding Dropouts + + + Adami 1995 Black 1996 ++ + + + + -- Bone 1997 + + + Chestnut 1995 Hosking 1998 + -- + + + + Liberman 1995 + + + McClung 1998 Quality Assessment: Example ++ indicates that randomization was appropriate ( eg Random numbers were computer generated)

19. Steps of a Cochrane Systematic Review • Well formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report

20. Outcome Discrete (event) Continuous (measured) Mean Standardized Difference Mean Difference (MD) (SMD) Odds Relative Risk Ratio Risk Difference (OR) (RR) (RD) (Basic Data) (Basic Data) Overall Estimate Fixed Effects Random Effects Overall Estimate Fixed Effects Random Effects

21. Effect measures: discrete data P1 = event rate in experimental group P2 = event rate in control group • RD = Risk difference = P2 - P1 • RR = Relative risk = P1 / P2 • RRR = Relative risk reduction = (P2-P1)/P2 • OR = Odds ratio = P1/(1-P1)/[P2/(1-P2)] • NNT = No. needed to treat = 1 / (P2-P1)

22. Example Experimental event rate = 0.3 Control event rate = 0.4 RD = 0.4 - 0.3 = 0.1 RR = 0.3 / 0.4 = 0.75 RRR = (0.4 - 0.3) / 0.4 = 0.25 OR = (0.3/0.7)/(0.4/0.6) = 0.64 NNT = 1 / (0.4 - 0.3) = 10

23. Discrete - Odds Ratio (OR) Event No event Experimental a b ne Control c d nc Odds: number of patients experiencing event number of patients not experiencing event Odds ratio: Odds in Experimental group Odds in Control group Basic Data a/ne c/nc

24. Discrete - Odds Ratio Example Event No event Experimental 13 33 46 Control 7 31 38 Basic Data 13/467/38

25. Discrete - Relative Risk (RR) Event No event Experimental a b ne Control c d nc Risk: number of patients experiencing event number of patients Risk Ratio: Risk in Experimental group Risk in Control group Basic Data a/ne c/nc

26. Discrete - Relative Risk - Example Event No event Experimental 13 33 46 Control 7 31 38 Basic Data 13/467/38

27. Discrete - Risk Difference (RD) Event No event Experimental a b ne Control c d nc Risk: number of patients experiencing event number of patients Risk Difference: (Risk in Experimental group) - (Risk in Control group) RD = Pe- Pc Basic Data a/ne c/nc

28. Discrete - Risk Difference - Example Event No event Experimental 13 33 46 Control 7 31 38 RD = Pe- Pc = 13/46 - 7/38 = 0.098 Basic Data 13/467/38

29. Discrete - Odds Ratio (O) Event No event Experimental a b ne Control c d nc Estimator: Standard Error: 100(1- )% CI:

30. Discrete - Relative Risk (R) Event No event Experimental a b ne Control c d nc Estimator: Standard Error: 100(1- )% CI:

31. Discrete - Risk Difference (D) Event No event Experimental a b ne Control c d nc Estimator: Standard Error: 100(1- )% CI:

32. When to use OR / RR / RD OR vs RR Odds Ratio  Relative Risk if event occurs infrequently (i.e. a and c small relative to b and d) RR = a(c+d)  ad = OR (a+b)c bc Odds Ratio > Relative Risk if event occurs frequently RD vs RR When interpretation in terms of absolute difference is better than in relative terms (eg. Interest in absolute reduction in adverse events)

34. Continuous Data - Mean Difference (MD) number mean standard deviation Experimental ne se Control nc sc

35. Continuous Data - Standardized Mean Difference (SMD) number mean standard deviation Experimental ne se Control nc sc

36. When to use MD / SMD Mean Difference • When studies have comparable outcome measures (ie. Same scale, probably same length of follow-up) • A meta-analysis using MDs is known as a weighted mean difference (WMD) Standardized Mean Difference • When studies use different outcome measurements which address the same clinical outcome (eg different scales) • Converts scale to a common scale: number of standard deviations

37. Example: Combining different scales for Swollen Joint Count

38. Sources of Variation over Studies • “True” inter-study variation may exist (fixed/random-effects model) • Sampling error may vary among studies (sample size) • Characteristics may differ among studies (population, intervention)

39. Modelling Variation • Parameter of interest: (quantifies average treatment effect) • Number of independent studies: k • Summary Statistic: Yi (i=1,2,…,k) • Large sample size: asymptotic normal distribution Fixed-effects model vs Random-effects model

40. Fixed-Effects Model • Outcome Yi from study i is a sample from a distribution with mean (ie. common mean across studies) • Yi are independently distributed as N ( , ) (i=1,2,…,k) where = Var(Yi ) and assume E(Yi) =

41. Fixed-Effects Model x

42. Random-Effects Model • Outcome Yi from study i is a sample from a distribution with mean (ie. study-specific means) • Yi are independently distributed as N ( , ) (i=1,2,…,k) where = Var(Yi ) and assume E(Yi) = • is a realization from a distribution of ‘effects’ with mean • are independently distributed as N ( , ) (i=1,2,…,k) where • = Var ( ) is the inter-study variation • is the average treatment effect

43. Random-Effects Model x

44. Random-Effects Model ….. • after averaging study-specific effects, distribution of Yi is N ( , ) • although is parameter of interest, must be considered and estimated Estimating Average Study Effect Estimating Study-Specific Effects • distribution of conditional on observed data, and is N ( ) • where Fi is the shrinkage factor for the ith study

45. Modelling Variation • Studies are stratified and then combined to account for differences in sample size and study characteristics • A weighted average of estimates from each study is calculated • Question of whether a common or study-specific parameter is to be estimated remains …. Procedure: • perform test of homogeneity • if no significant difference use fixed-effects model • otherwise identify study characteristics that stratifies studies into subsets with homogeneous effects or use random effects model

46. Fixed Effects Model • Require from each study • effect estimate; and • standard error of effect estimate Combine these using a weighted average: pooled estimate = sum of (estimate  weight) sum of weights where weight = 1 / variance of estimate • Assumes a common underlying effect behind every trial

47. Fixed-Effects Model: General Scheme Study Measure Std Error Weight 1 Y1 s1 W1 2 Y2 s2 W2 . . . . . . . . . . . . k Yk sk Wk (no association: Yi=0) Overall Measure:

48. Chi-Square Tests: 1 2 1 If ‘large’ association 2 If ‘large’ heterogeneity

49. Features in Graphic Display • For each trial • estimate (square) • 95% confidence interval (CI) (line) • size (square) indicates weight allocated • Solid vertical line of ‘no effect’ • if CI crosses line then effect not significant (p>0.05) • Horizontal axis • arithmetic: RD, MD, SMD • logarithmic: OR, RR • Diamond represents combined estimate and 95% CI • Dashed line plotted vertically through combined estimate

50. Odds Ratio Three methods for combining (1) Mantel-Haenszel method (2) Peto’s method (3) Maximum likelihood method Relative Risk Risk Difference