Introduction to Robust Standard Errors

1. Introduction to Robust Standard Errors Emily E. Tanner-Smith Associate Editor, Methods Coordinating Group Research Assistant Professor, Vanderbilt University Campbell Collaboration Colloquium Copenhagen, Denmark May 30th, 2012

2. Outline • Types of dependencies • Dealing with dependencies • Robust variance estimation • Practical considerations • Choosing weights • Handling covariates • Robust variance estimation in Stata

3. Types of Dependencies • Most meta-analysis techniques assume effect sizes are statistically independent • But there are many instances when you might have dependent effect sizes • Multiple measures of the same underlying outcome construct • Multiple measures across different follow-up periods • Multiple treatment groups with a common control group • Multiple studies from the same research laboratory

4. Types of Dependencies • Assume Ti = θi + εi, where • Ti is the effect size estimate • θi is the effect size parameter • εi is the estimation error • Statistical dependence can arise because • εi are correlated • θi are correlated • or both

5. Correlated Effects Model (εi correlated)

6. Hierarchical Model (θi correlated)

7. Dealing with Dependencies • Ignore it and analyze the effect sizes as if they are independent (not recommended) • Select a set of independent effect sizes • Create a synthetic mean effect size • Randomly select one effect size • Choose the “best” effect size • Model the dependence with full multivariate analysis • This requires information on the covariance structure • Use robust variance estimation (Hedges, Tipton, & Johnson, 2010)

8. Variance Estimation • Assume T = Xβ + ε, where

9. Variance Estimation • We can estimate β by: • And the covariance matrix for this estimate is • The problem is that although the variances in Σj are known, the covariances are UNKNOWN

10. Robust Variance Estimator (RVE) • The RVE of b is where ej = Tj – Xjb is the kj x 1 estimated residual vector for study j

11. Robust Variance Estimator (RVE) • A robust test of H0: βa= 0 uses the statistic where vRaa is the ath diagonal of the VR matrix Note: the t-distribution with df = m - p should be used for critical values

12. Robust Variance Estimator (RVE) • Under regularity conditions and as m -> ∞, VmR is a consistent estimator of the true covariance matrix • RVE theorem is asymptotic in the number of studies m, not the number of effect sizes • Results apply to any type of dependency • No distributional assumptions needed for the effect sizes • Correlations do not need to be known or specified, though may impact the standard errors • RVE theorem applies for any set of weights

13. Practical Issues: Choosing Weights • Although the RVE works for any weights, the most efficient weights are inverse-variance weights • In the hierarchical model:

14. Practical Issues: Choosing Weights • In the correlated effects model, we can estimate approximately efficient weights by assuming a simplified correlation structure: • Within each study j, the correlation between all pairs of effect sizes is a constant ρ • ρ is the same in all studies • kj sampling variances within the study are approximately equal with average Vj

15. Practical Issues: Choosing Weights • In the correlated effects model, we can take a conservative approach when calculating weights by also assuming ρ = 1, and weights become: • Conservative approach, because studies do not receive additional weight for contributing multiple effect sizes

16. Practical Issues: Choosing Weights • In the correlated effects model, ρ also occurs in the estimator of τ2: • Use external information about ρif available (test reliabilities, correlations reported in studies, etc.) • Take a sensitivity approach when estimating τ2by estimating the model with various values of ρ in (0, 1)

17. Practical Issues: Choosing Weights • For the correlated effects model, RVE software is currently programmed to default to (per Hedges, Tipton, & Johnson recommendation): • Conservative approach to estimate weights (assume ρ = 1) • User must specify ρfor estimation of τ2; sensitivity tests recommended

18. Practical Issues: Handling Covariates • Some covariates may vary withingroups (i.e., studies or clusters) and betweengroups, e.g., • Length of follow-up after intervention • Time frame of outcome measure • Outcome reporter (self-report vs. parent-report) • Type of outcome construct (frequency vs. quantity of alcohol use) • When modeling the effects of a covariate, ask if the effect of interest is between- or within-groups

19. Practical Issues: Handling Covariates • In a standard meta-regression with independent effect sizes, Tj = β0 + Xjβ1 + … where Xj is length to follow-up, β0 and β1 can be interpreted as: • β0 = the average effect size when Xj = 0 • e.g. the average effect size in studies in which the intervention just occurred • β1 = the effect of a 1-unit increase in Xj on Tj • e.g. the effect size change associated with moving from a study in which the intervention just occurred to a study in which the effect size was measured at a 1 month posttest follow-up

20. Practical Issues: Handling Covariates • In the correlated effects model, for a fixed study (j = 1), now assume there are multiple outcomes. This study has its own regression equation: Ti1 = β01 + Xi1β11 +… • The coefficients β01 and β11 can be interpreted as: • β01 = the average effect size when Xi1 = 0 • e.g. the average effect size for units in the study (j = 1) when the intervention just occurred • β11 = the effect of a 1-unit increase in Xi1 on Ti1 • e.g. the effect size change for units in the study (j = 1) at the time of intervention and at follow-up 1 month later

21. Practical Issues: Handling Covariates • When using the RVE, these two types of regression occur in one analysis: Within GroupTij= β0j + Xijβ2 + … Between Group β0j = β0 + Xjβ1 + … • These two regressions are combined into one analysis and model: Tij= β0 + Xijβ2 + Xjβ1 + …

22. Practical Issues: Handling Covariates • To properly separate estimation of within- and between-group effects of covariates, use group mean centering: Xcij= Xij – Xj where Xj is the mean value of Xij in group j (and where group is either study or cluster). So now, Tij = β0 + Xcijβ2+ Xjβ1 + … • If you don’t center Xij you are actually modeling a weighted combination of the within- and between-study effect, which is difficult to interpret

23. Practical Issues: Handling Covariates • When using a covariate, ask if the effect of interest is between- or within-groups • Make sure to group-center your within-group variables • Acknowledge that ifonly a few groups have variability in Xij • Within-group estimate of β2 (associated with Xcij) will be imprecise (i.e. have a large standard error) • The types of groups in which Xij varies may be different than (i.e. not representative of) groups in which Xij does not vary

24. Calculating Robust Variance Estimates • Variables you will need in your dataset • Group identifier (e.g., study/cluster identification number) • Effect size estimate • Variance estimate of the effect size • Any moderator variables or covariates of interest • Additional pieces of information you will need to specify • In a correlated effects model: assumed correlation between all pairs of effect sizes (ρ) • Fixed, random, hierarchical, or user-specified weights

25. Calculating Robust Variance Estimates • Stata ado file available at SSC archive: type ssc install robumeta • SPSS macro available at: http://peabody.vanderbilt.edu/peabody_research_institute/methods_resources.xml • R functions available at: http://www.northwestern.edu/ipr/qcenter/RVE-meta-analysis.html

26. Robust Variance Estimation in Stata • Install robumeta.ado file from the Statistical Software Components (SSC) archive

27. Robust Variance Estimation in Stata • Access example datasets and syntax/output files here: https://my.vanderbilt.edu/emilytannersmith/training-materials/

28. Robust Variance Estimation in Stata • robumeta.ado command structure robumeta depvar[indepvars] [if] [in], study(studyid) variance(variancevar) weighttype(weightingscheme) rho(rhoval) [options]

29. Robust Variance Estimation in Stata • Example of a correlated effects model (correlated ε) • Fictional meta-analysis on the effectiveness of alcohol abuse treatment for adolescents • Effect sizes represent post-treatment differences between treatment and comparison groups on some measure of alcohol use (positive effect sizes represent beneficial treatment effects) • Number of effect sizes k = 172 • Number of studies m = 39 • Average number of effect sizes per study = 4.41

30. Robust Variance Estimation in Stata

31. Robust Variance Estimation in Stata Intercept only model to estimate random-effects mean effect size with robust standard error, assuming ρ = .80

32. Robust Variance Estimation in Stata

33. Robust Variance Estimation in Stata

34. Robust Variance Estimation in Stata • 4 moderators of interest: 2 vary within and between studies, 2 vary between studies only

35. Robust Variance Estimation in Stata • To model both the within- (Xcij) and between- effects (Xj) of the type of alcohol outcome and follow-up time frame, create group mean and group mean centered variables

36. Robust Variance Estimation in Stata

37. Robust Variance Estimation in Stata • Let’s say we have a similar meta-analysis, but now need to estimate a hierarchical model (correlated θ) • Effect sizes represent post-treatment differences between treatment and comparison groups on some measure of alcohol use (positive effect sizes represent beneficial treatment effects) • Number of effect sizes k = 68 • Number of research labs m = 15 • Average number of effect sizes per research lab = 4.5

38. Robust Variance Estimation in Stata τ2– between lab variance component; ω2 between-study within-lab variance component

39. Robust Variance Estimation in Stata • 5 moderators of interest: all vary within and between clusters (research labs)

40. Robust Variance Estimation in Stata • To model both the within- (Xcij) and between- effects (Xj) of the covariates of interest, create group mean and group mean centered variables

42. Conclusions & Recommendations • Robust variance estimation is one way to handle dependencies in effect size estimates, and allows estimation of within- and between-study effects of covariates • Method performs well when there are 20 or more studies with an average of 2 or more effect size estimates per study • Choose the proper model for the type of dependencies in your data (correlated ε or correlated θ)

43. Conclusions & Recommendations • When using the correlated effects model (correlated ε), with efficient weights, if you have no information on ρ: • Use a sensitivity approach for estimating τ2 • Assume ρ= 1 in your weights, i.e., • For each covariate Xij in your model, remember that you can estimate: • Between-group effect: group mean (Xj) • Within-group effect: group mean centered variable (Xcij= Xij – Xj)

44. Recommended Reading Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39-65.

45. P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway E-mail: info@c2admin.org http://www.campbellcollaboration.org