Download
1 / 14
140 likes | 158 Views
Meta-analysis in R with Metafor. Sensitivity Analysis. Examining outcomes under different decisions. Quality of Studies. We make inclusion decisions and worry about the quality of the studies included in the analysis.
E N D
Meta-analysis in R with Metafor Sensitivity Analysis Examining outcomes under different decisions
Quality of Studies • We make inclusion decisions and worry about the quality of the studies included in the analysis. • Both inclusion decisions and study quality can be treated as a moderator if you have the data • moderators were covered in an earlier video.
Models and Data • Parameter estimates can be sensitive to influential data points • Can be deviant in the sense of extreme distance from the mean or the regression line • Can be very large sample • Missing studies (availability bias) • Outlier detection and removing studies • Leave-one-out
Sensitivity Analyses • Forest plot by precision (trim-and-fill covered in funnel plot video) • Egger’s regression (statistical test of asymmetry) • Residuals • Mean only • Moderator in model • Leave-one-out • Statistics (Mean, I-squared, etc.) • Graph (plot the mean and CI excluding each ES)
Forest Sleep & AD Note how the small studies are mostly all on the bottom right. This suggests availability bias.
Egger’s Regression Ti = effect size; vi=sampling variance of ES Should be flat (centered) if no bias (beta1 is zero). This shows small studies have higher values. Significant negative slope (beta1) is concerning. Source: Sutton (2009). In Cooper, Hedges, & Valentine (Eds) Handbook fo Research Synthesis Methods p. 441
Funnel Asymmetry Test Metafor offers several different tests for funnel plot asymmetry. The regtest results are shown below for the sleep data shown previously.
Finding Outliers This is the result of an overall meta-analysis with no moderators – it is the overall result for the varying (random) effects analysis. Equivalent to a regression with intercept only.
Finding Outliers (3) This analysis considers the study sample size and the REVC in addition to the raw distance to the mean. Look for ‘large’ values – the z refers to the unit normal, so |2| is large, but I would probably start with 2.5 or 3. Your call though. I would sort by z (there are 92 residuals in this analysis). LMX data
Outliers 4 Here I have inserted a categorical independent variable into the model. The residuals are now studentized, deleted residuals that remain after accounting for differences in culture (vertical integration and collective orientation). As you can see, adding the moderator made very little difference for observation 12 (-2.08 to -2.05). For an important moderator, there could be a large difference.
Removing Observations Remove one outlier negative sign to take out, no negative sign, include only those. Remove 2 outliers, run additional analyses
If you show that the impact of outliers and nuisance variables (and moderators) is minimal or at least not a threat to your inferences, then your conclusions will be more credible.
Leave One Out • In this sensitivity analysis, every study is removed, 1 by 1, and the overall ES is re-estimated. • Helpful to judge the impact of influential studies (sometimes outlying ES, sometimes very large N) • Can be run in rma if the model has no moderators (i.e., for overall ES or a subset of studies) • Two uses for this • Find and remove problematic studies • Share summary of findings with your reader
McNatt Example • R-code for McNatt
