1 / 92

Systematic Review: Analytical Methods of Meta-analysis

Systematic Review: Analytical Methods of Meta-analysis. Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF. 8 Steps to Systematic Review. 1. Research Question 2. Protocol 3. Search 4. Study selection (inclusion/exclusion) 5. Quality assessment

gary
Download Presentation

Systematic Review: Analytical Methods of Meta-analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Systematic Review:Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF

  2. 8 Steps to Systematic Review • 1. Research Question • 2. Protocol • 3. Search • 4. Study selection (inclusion/exclusion) • 5. Quality assessment • 6. Data abstraction • 7. Analysis • A) Create summary measure • B) Assess for heterogeneity • C) Assess for publication bias • D) Conduct sensitivity/subgroup analyses • E) Advanced issues/techniques • 8. Interpretation

  3. Juni et al, Hazards of scoring the quality of clinical trials. JAMA. 1999;282:1054-60.

  4. Why conduct a systematic review? • The best way to summarize evidence on a scientific topic • Concisely communicates findings to others in the field • Identifies author(s) as experts • Identifies areas for future study • Perfect for background of grants • Don’t need to do primary data collection (so can be done while waiting for data from other projects) • You have to do the work anyway, so might as well get a publication! • You can effect change in clinical management

  5. Cumulative Meta-analysis Antman EM et al: JAMA. 1992;268:240-248

  6. Systematic Review: Clinical Implications (Antiarrhythmic Drugs for Acute MI) Teo KK et al. JAMA. 1993;270:1589-1595

  7. Sample Systematic Reviews • Kangelaris KN, Bent S, Nussbaum RL, Garcia DA, Tice JA. Genetic testing before anticoagulation? A systematic review of the safety and efficacy of pharmacogenetic dosing of warfarin. Journal of General Internal Medicine (in press). • Nguyen SP, Bent S, Chen Y, Terdiman JP. Gender as a Risk Factor for Advanced Neoplasia and Colorectal Cancer: A Systematic Review and Meta-analysis. Clinical Gastroenterology. 2009;7:676-81. • Simon J, Chen Y, Bent S. The relation of alpha-linoleic acid to the risk of prostate cancer: a systematic review. Am J Clin Nutr. 2009;89:1-7S. • Li J, Winston LG, Moore DH, Bent S. Efficacy of short-course antibiotic regimens for community-acquired pneumonia: a meta-analysis. American Journal of Medicine. 2007;120(9):783-90. • Margaretten M, Kohlwes J, Moore D, Bent S. The rational clinical examination: does this patient have septic arthritis. JAMA. 2007;297:1478-1488.

  8. Sample Systematic Reviews Hsu J, Kohlwes J, Bent S. Efficacy of antifungal therapy in chronic rhinosinusitis: A systematic review. J Allergy Clin Immunol. 2010 125:2 Guarnieri M, Bent S. Death from coronary artery disease in patients with systemic lupus erythematosus: a systematic review and meta-analysis of mortality cohort studies. (submitted to Arthritis Care and Research 1/2012). Lee JK, Liles EG, Bent S, Levin TR, Corley DA. Diagnostic Accuracy of Fecal Immunochemical Tests for Colorectal Cancer: Systematic Review and Meta-analysis (submitted to JAMA 4/2013).

  9. 8 Steps of Systematic Review • 1. Research Question • 2. Protocol • 3. Search • 4. Study selection (inclusion/exclusion) • 5. Quality assessment • 6. Data abstraction • 7. Analysis • A) Create summary measure • B) Assess for heterogeneity • C) Assess for publication bias • D) Conduct sensitivity/subgroup analyses • E) Advanced issues/techniques • 8. Interpretation

  10. Create a Summary Measure • Before we get to the mechanics of a summary measure…. • Be sure to provide your audience with a concise summary table • A “visual meta-analysis” • Readers should be able to examine Table 1 and reach their own conclusions about the data

  11. Example • Antibiotics for acute bronchitis. • After search and application of inclusion/exclusion criteria, 8 studies were included.

  12. RCTs in Acute Bronchitis * Positive numbers indicate antibiotics are superior to placebo

  13. How do you create a summary measure? • Clinical example: 5 year old girl presents with ear pain and is found to have an acute otitis media. • Should she get antibiotics? Research Questions: • In children with OM, are antibiotics effective for pain relief? • In children with OM, do antibiotics reduce the rate of complications (mastoiditis, hearing problems)?

  14. 3 studies are identified (examining effect of Abx on Pain) • Study 1: N = 100 RR=1.41 • Study 2: N=200 RR=0.98 • Study 3: N=300 RR=1.01 • You could take the average effect: (1.41 + 0.98 + 1.01) / 3 = 1.13 • Is this a good summary measure?

  15. Study N RR 1 100 1.41 2 200 0.98 3 300 1.01 Total 600 Summary measure weighted by sample size • Provide “weight” for studies based on their sample size summary effect estimate= Σ (Ni x effect estimatei) = 640 =1.07 Σ(Ni) 600

  16. More refined: Provide “weight” by using inverse of variance Summary = Σ (weighti x effect estimatei) = 30.5 = 1.00 effect estimate Σ(weighti) 30.3

  17. Does the largest study always have the lowest variance and therefore the greatest “weight”? • Dichotomous outcomes • Continuous outcomes

  18. Confidence Intervals Around Summary Effect • Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weightsi) Variance of summary estimate = _1_ = .03 30.5 95% CI = + 1.96 √0.03 = + 0.34 Summary OR and 95% CI = 1.00 (0.65 - 1.33)

  19. Fixed Effects Random Effects Goal: weighted average of risk from existing studies Goal: estimate the “true” effect Existing studies are the entire population Existing studies are a random sample Weights: variance of individual studies Variance of individual studies + variance of differences between studies Weighti = 1 variance RRi Weighti = 1 variance RRi + D Variance RRs = 1/wi Variance RRs = 1/wi Type of Model?

  20. Formulas for D

  21. a Summary RR b Summary RR a Summary RR b Summary RR Fixed Effects Model: Random Effects Model:

  22. Random VS. Fixed Effects Model Practical Implications of the Choice • Confidence intervals: RE model produces wider confidence intervals • Statistical significance: less likely with RE model BOTTOM LINE: • If the individual study findings are similar, the model makes little difference in estimate or statistical significance. • If the individual study findings are heterogeneous, the model can affect the statistical significance.

  23. Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) ai ci Not treated (unexposed) bi di ORi = ai/ci = ai x dilnORmh = Σ (wi x lnORi ) bi/di bix ci Σwi variance lnORi = 1 + 1 + 1+ 1 variance ORmh = 1 ai bi ci di Σ wi weighti = (wi) = 1 variance lnORi 95% CI = elnORmh (1.96 x √variance lnORmh)

  24. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1PerforationNo Perforation Antibiotic 1 114 Placebo 3 116 Study 2PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 1. Calculate OR and lnOR for each study: OR1= 1 x 116 = 0.34 lnOR1 = -1.08 3 x 114 OR2 = 7 x 65 = 0.58 lnOR2 = -0.54 12 x 65

  25. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnORi for each study: Varln OR1 = 1 + 1 + 1 + 1 = 1.35 1 3 114 116 Var ln OR2 = 1 + 1 + 1 + 1 = 0.26 7 12 65 65 3. Calculate wi for each study: w1 = 1 = 0.74 1.35 w2 = 1 = 3.85 0.26

  26. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1PerforationNo Perforation Antibiotic 1 114 Placebo 3 116 Study 2PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 4. Calculate the wi x ln ORi for each study: w1 x lnOR1 = 0.74 x -1.08 = -0.80 w2 x lnOR2= 3.85 x -0.54 = -2.08

  27. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 5. Calculate the sum of the wi w1 + w2 = 0.74 + 3.85 = 4.59 • Summary lnORmh =Σ (wi x lnORi) = -0.80 + -2.08 = -0.63Σ wi 4.59 = ORmh = 0.53 • Calculate variance ORmh = 1 = 1 = 0.22 Σ wi 4.59 8. Calculate 95% CI = elnORmh + (1.96 x √ variance lnORmh) = e-.63 + (1.96 x √ 0.22) = 0.21 - 1.34 Summary OR = 0.53 (95% CI 0.21 – 1.34)

  28. Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln ORdl = Σ (wix ln ORi) wi = 1 Σwi variancei + D • Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

  29. But…All you need to know is: • When combined, individual study effect estimates are weighted by their inverse variance • Variance is related to sample size AND # of events (dichotomous) and precision (continuous) • Fixed effects just combines all weighted estimates, while random effects “penalizes” estimates for variation between studies

  30. 8 Steps to Systematic Review • 1. Research Question • 2. Protocol • 3. Search • 4. Study selection (inclusion/exclusion) • 5. Quality assessment • 6. Data abstraction • 7. Analysis • A) Create summary measure • B) Assess for heterogeneity • C) Assess for publication bias • D) Conduct sensitivity/subgroup analyses • E) Advanced issues/techniques • 8. Interpretation

  31. Heterogeneity Are you comparing apples and oranges? Clinical heterogeneity: are studies asking same question? Statistical heterogeneity: is the variation likely to have occurred by chance? Measures how far each individual OR/RR is from the summary OR/RR. Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

  32. Heterogeneity • Refers to the degree that the study results differ • Visual Approach • Statistical Approach Q = sum [weighti x (ESs – ESi)] p < 0.05 indicates heterogeneity

  33. Summary RR = 0.93 (0.87-0.99)

  34. Problem of Heterogeneity • Study findings are different – should they be combined? StudyOR 1 0.01 2 1.0 3 10.0 StudyOR 1 0.35 2 0.56 3 0.97 4 1.15 5 1.75 6 1.95

  35. Statistical tests of Heterogeneity • Is the variation in the individual study findings likely due to chance? Ho: Effect estimate in each study is the same (or homogeneous) Ha: Effect estimate in each study is not the same (or heterogeneous) Q = Σ(wi x (ln ORmh – ln ORi )2) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous

  36. Heterogeneity – Interpret Findings (Example: RR of Colon CA, Men vs. Women)

  37. 8 Steps to Sytematic Review • 1. Research Question • 2. Protocol • 3. Search • 4. Study selection (inclusion/exclusion) • 5. Quality assessment • 6. Data abstraction • 7. Analysis • A) Create summary measure • B) Assess for heterogeneity • C) Assess for publication bias • D) Conduct sensitivity/subgroup analyses • E) Advanced issues/techniques • 8. Interpretation

  38. Assessing for Publication Bias • Publication Bias – the publication or “non-publication” of research findings, depending on the nature and direction of the results. • Rosenthal, 1979 – published an article describing the “file-drawer problem” that journals publish only 5% of all negative studies, while the file drawers in the back of the lab contain the other 95%.

  39. Methods for Assessing Publication Bias • Funnel plots – simple scatter plots of treatment effects (horizontal axis) vs. some measure of study size (vertical axis). • Choice of axes • Log scale for treatment effects (to ensure that treatment effects in opposite directions are the same distance from 1.0 – e.g., 0.5 and 2.0) • Standard error for measure of sample size • Power depends on both sample size and # events • Standard error is consistent with the statistical tests

  40. Funnel Plot of Log Relative Risk vs Standard Error 0 1 2 Standard error 3 4 5 Log Relative Risk

  41. Example: ALA and Prostate Cancer Risk RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

  42. ALA – Funnel Plot

  43. Funnel Plot with Imputed Values for Publication Bias RR=0.94, 95% CI: 0.79-1.17

  44. Publication bias caveat • Funnel plot asymmetry does not always indicate bias • It is possible that smaller studies enrolled higher risk patients, for example, and therefore found a greater effect. • Small studies are often conducted before larger studies. In the intervening years, other interventions may have improved, thus reducing the relative efficacy of the treatment.

  45. Statistical methods to assess publication bias • Examine associations between study size and treatment effect. • Sensitivity is poor when < 20 studies • Begg’s test: an adjusted rank correlation • Egger’s test: a weighted regression of effect size vs. standard error. • Basically asks if the regression line has a non-zero slope • More sensitive than Begg’s test, but more false positives, especially when 1) large treatment effects, 2) few events per trial, 3) all trials of similar size. (In these cases, one may decide a priori to use Begg’s test).

  46. Begg's Test adj. Kendall's Score (P-Q) = -30 Std. Dev. of Score = 14.58 Number of Studies = 12 z = -2.06 Pr > |z| = 0.040 z = 1.99 (continuity corrected) Pr > |z| = 0.047 (continuity corrected) Egger's test ------------------------------------------------------------------------------ Std_Eff | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- slope | .9810716 .1103858 8.89 0.000 .7351168 1.227026 bias | -.9911295 .3236382 -3.06 0.012 -1.71224 -.2700187 ------------------------------------------------------------------------------

  47. 8 Steps to Systematic Review • 1. Research Question • 2. Protocol • 3. Search • 4. Study selection (inclusion/exclusion) • 5. Quality assessment • 6. Data abstraction • 7. Analysis • A) Create summary measure • B) Assess for heterogeneity • C) Assess for publication bias • D) Conduct sensitivity/subgroup analyses • E) Advanced issues/techniques • 8. Interpretation

More Related