Methods for Summarizing the Evidence: Meta-Analyses and Pooled Analyses

1. Methods for Summarizing the Evidence: Meta-Analyses and Pooled Analyses Donna Spiegelman, Sc.D. Departments of Epidemiology and Biostatistics Harvard School of Public Health stdls@channing.harvard.edu

2. Methods for Summarizing the Evidence • Narrative review • Meta-analysis of published data • Pooled analysis of primary data (meta-analysis of individual data) • Retrospectively planned • Prospectively planned

3. Narrative Review

4. Fruits, Vegetables and Breast Cancer: Summary of Case-Control Studies Total number of studies: 12 Studies showing a statistically significant protective association: 8 (67%) Studies showing no statistically significant protective association: 4 (33%) WCRF/AICR 1997

5. Specific Fruits and Vegetables and Breast Cancer: Summary of Case-Control Studies # of % of Total Food Group studies risk null risk Fruits 4 75 0 25 Citrus Fruits 3 33 0 66 Vegetables <3 -- -- -- Green Veg. 6 83 17 0 Carrots 4 75 25 0 WCRF/AICR 1997

6. Fruits, Vegetables and Breast Cancer: Conclusion of Narrative Review A large amount of evidence has accumulated regarding vegetable and fruit consumption and the risk of breast cancer. Almost all of the data from epidemiological studies show either decreased risk with higher intakes or no relationship; the evidence is more abundant and consistent for vegetables, particularly green vegetables, than for fruits. Diets high in vegetables and fruits probably decrease the risk of breast cancer. WCRF/AICR 1997

7. Narrative Review • Strengths • Short timeframe • Inexpensive • Limitations • Provide qualitative summary only frequently tabulate results • Subjective • Selective inclusion of studies • May be influenced by publication bias

8. Publication Bias: Funnel Plots Sterne 2001

9. Where Can Publication Bias Occur? • Project dropped when preliminary analyses suggest results are negative • Authors do not submit negative study • Results reported in small, non-indexed journal • Editor rejects manuscript • Reviewers reject manuscript • Author does not resubmit rejected manuscript • Journal delays publication of negative study • Results not reported by news, policy makers, or narrative reviews Montori 2000

10. Meta-Analyses

11. Rationale for Combining Studies “Many of the groups…are far too small to allow of any definite opinion being formed at all, having regard to the size of the probable error involved.” Pearson, 1904

12. Definition “Meta-analysis refers to the analysis of analyses…the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding literature…” Glass, 1976

13. Meta-Analyses: # of Publications by Year

14. Why conduct a meta-analysis? • Summarize published literature • A more objective summary of literature than narrative review • Estimate average effect from all available studies • Increase statistical power more precise estimate of effect size • Identify between study heterogeneity • Identify research needs

15. Outline for Conducting Meta-Analyses • Objective, hypothesis • Define outcome, exposure, population • Study inclusion criteria • Search strategy • Data extraction • Quality assessment • Statistical analysis

16. Meta-Analyses: Study Sources • Published literature • citation indexes • abstract databases • reference lists • contact with authors • Unpublished literature • Uncompleted research reports • Work in progress

17. Meta-Analyses: Data Extraction • Publication year • Performing year • Study design • Characteristics of study population (n, age, sex) • Geographical setting • Assessment procedures • Risk estimate and variance • Covariates

18. Meta-Analyses: Quality Assessment • Study components • Study design • Outcome measurement • Exposure measurement • Response rate/follow-up rate • Analytic strategy • Adjustment for confounding • Quality of reporting

19. Meta-Analyses: Statistical Analyses • Define analytic strategy • Investigate between study heterogeneity • Decide whether results can be combined • Estimate summary effect, if appropriate • Conduct sensitivity analysis • Stratification • Meta-regression analyses

20. Fixed Effects • Assumes that all studies are estimating the same true effect • Variability only from sampling of people within each study • Precision depends mainly on study size

21. Fixed Effects Model

22. Random Effects • Studies allowed to have different underlying or true effects • Allows variation between studies as well as within studies

23. Random Effects Model

24. Random Effects Model, continued

25. Test for Heterogeneity

26. Fixed Effects vs Random Effects Model • Random effects generally yield larger variances and CI • Why? Incorporate • If heterogeneity between studies is large, will dominate the weights and all studies will be weighted more equally • Model weight for large studies less in random vs fixed effects model

27. Sources of Between Study Heterogeneity • Different study designs • Different incidence rates among unexposed • Different length of follow-up • Different distributions of effect modifiers • Different statistical methods/models used • Different sources of bias • Study quality

28. Meta-Analyses: Sensitivity Analyses • Exclude studies with particular heterogeneous results • Conduct separate analyses based on • Study design • Geographic location • Time period • Study quality

29. Meta-regression (Stram, Biometrics, 1996) Purpose: to identify heterogeneity of effects by covariates that are constant within study (e.g. gender, smoking status) Model: β s = β 0 + β1 GENDERs+ β2CURRENTs + β2PASTs + bs + εs GENDERs = 1 if studysis male; 0 if female CURRENTs = 1 if studyshas current smokers only, 0 otherwise PASTs= 1 if studyshas past smokers only, 0 otherwise H0: β1 = 0  no effect-modification by gender Standard method for mixed effects models can be used to test hypotheses and estimate parameters ^

30. Analysis of between-studies heterogeneity • p-value for test for heterogeneity is a function of the power of the pooled analysis to detect between-studies differences. This power is believed to be low. • a simulation study was conducted and published (Takkouche, Cardoso-Suárez, Spiegelman, AJE, 1999) which investigated the power of several old and some newly developed test statistics to detect heterogeneity of different plausible magnitudes, as quantified by CV and R2 (to be defined) • CV = /Ι β Ι , with S=ranging from 7 to 33 12

31. Analysis of between-studies heterogeneity • Explored maximum likelihood methods for estimation of and testing H0: =0 (i.e. no between-studies heterogeneity) • ML methods have power roughly equivalent to D&L’s, but assume bs ~N (0, ) and εs ~N [ 0, Vars ( βs ) ] • LRT for H0: = 0 has no known asymptotic distribution because hypothesis is on the boundary of the parameter space • A simulation-based bootstrap approach for constructing the empirical distribution function of the test statistic was developed ^ 14

32. Quantification of heterogeneity CVB = /β • between-studies variance expressed relative to the magnitude of the overall association • if the association is small, CV 'blows up’ • proportion of the variance of the pooled estimate due to between-studies variation • Useful for when β is near 0 as well as when it is far from it • Heterogeneity is evaluated relative to within-studies contribution to the variance, and can appear large if the participating studies yield precise estimates • Further experience with these measures will give us more insight as to their relative merits • Confidence intervals for CVB, R2 (Identical to I2 (Higgins et al., 2006)) ^ 1

33. Power of Test of Heterogeneity * Parametric bootstrap version of the test. † Odds ratio = 2. ‡ WLS, weighted least squares; LRT, likelihood ratio test. § R2, proportion of the total variance due to between-study variance: /(( + (S x Var())). 13

34. Fruit & Vegetable and Breast Cancer Meta-Analysis • Objective: analyze published results that explore the relationship between breast cancer risk and the consumption of fruits and vegetables • Search strategy • MEDLINE search of studies published January 1982 – April 1997 • Review of reference lists Gandini, 2000

35. Fruit & Vegetable and Breast Cancer Meta-Analysis: Inclusion Criteria • Relative risks and confidence intervals reported or could be estimated • Comparisons: tertiles, quartiles, quintiles • Studies were independent • Diet assessed by food frequency questionnaire • Populations were homogeneous, not limited to specific subgroup Gandini, 2000

36. Fruit & Vegetable and Breast Cancer Meta-Analysis: Data Extraction and Analysis • Selected risk estimate for total fruits and total vegetables, when possible • Otherwise, selected nutrient dense food • Extracted the most adjusted relative risk comparing the highest vs. lowest intake • Comparisons: tertiles, quartiles, quintiles • Used random effects model to calculate summary estimate • Sensitivity analyses Gandini, 2000

37. Fruit and Vegetable and Breast Cancer Meta-Analysis: Results RR (95% CI) p for het high vs low Total fruits 0.94 (0.79-1.11) <0.001 Total vegetables 0.75 (0.66-0.85) <0.001 Gandini, 2000

38. Fruits and Vegetables and Breast Cancer Meta-Analysis: Sensitivity Analysis # of RR Significance Studies of factors Study design Case-control 14 0.71 Cohort 3 0.73 0.30 Validated FFQ Yes 6 0.85 No 11 0.66 0.13 Gandini, 2000

39. Fruits and Vegetables and Breast Cancer: Conclusion of Meta-Analysis • The quantitative analysis of the published studies…suggests a moderate protective effect for high consumption of vegetables… For fruit intake, study results were less clear. Only two studies show a significant protective effect of high fruit intake for breast cancer. • This analysis confirms the association between intake of vegetables and, to a lesser extent, fruits and breast cancer risk from published sources. Gandini, 2000

40. Limitations of Meta-Analyses • Heterogeneity across studies • Difficult to evaluate dose-response associations • Difficult to examine population subgroups • Errors in original work cannot be checked • Errors in data extraction • Limited by quality of the studies included • May be influenced by publication bias

41. Pooled Analyses

42. Outline for Conducting Pooled Analyses • Search strategy • Study inclusion criteria • Obtain primary data • Prepare data for pooled analysis • Estimate study-specific effects • Examine whether results are heterogeneous • Estimate pooled result • Conduct sensitivity analyses Friedenreich 1993

43. Pooling Project of Prospective Studies of Diet and Cancer • Collaborative project to re-analyze the primary data in multiple cohort studies using standardized analytic criteria to generate summary estimates • Retrospectively-planned meta-analysis of individual patient data • Established in 1991 • http://www.hsph.harvard.edu/poolingproject/about.html

44. Pooling Project of Prospective Studies of Diet and Cancer: Inclusion Criteria • Prospective study with a publication on diet and cancer • Usual dietary intake assessed • Validation study of diet assessment method • Minimum number of cases specific for each cancer site examined

45. Cohort Studies in the Pooling Project of Prospective Studies of Diet and Cancer Canadian National Breast Screening Study Sweden Mammography Cohort New York University Women’s Health Study Netherlands Cohort Study Health Professionals Follow-up Study, Nurses’ Health Study, Women’s Health Study, Nurses’ Health Study II CA Teacher’s Study New York State Cohort ORDET Alpha-Tocopherol Beta-Carotene Cancer Prevention Study Adventist Health Study Breast Cancer Detection Demonstration Project Cancer Prevention Study II Nutrition Cohort Iowa Women’s Health Study Total=948,983

46. Analytic Strategy Nutrients Main effect Food groups Study-specific analyses Data collection Population subgroups Pooling Foods Annual meeting Non-dietary risk factors Effect modification

47. Pooling Project: Data Management • Receive primary data • Different media • Different formats • Check data  follow-up with investigators • Apply exclusion criteria • Calculate energy-adjusted nutrient intakes • Create standardized name and format for each variable

48. Pooling Project: Data Checks • Dates • Questionnaire return • Follow-up time • Diagnosis • Death • Non-dietary variables • Frequency distributions, means • Consistency checks • Parity and age at first birth • Smoking status and # cigarettes • Nutrients and foods: means, range

49. Data Management: Inconsistent UnitsExample: Vitamin A, Carotene • Units: • g retinol equivalents • g • International units • How determine units • Original data sheets • Published values • Correspondence

50. Data Management: Standardizing Food Data • Created standardized name for each food on the FFQs (range: 46-276 items) • Multiple foods on same line • “Other” categories • Standardized quantity information as grams/d • For some studies, calculated gram intakes by frequency * portion size * gram weight • Defined food groups