The Basics of Effect Size Extraction and Statistical Applications for Meta-Analysis

1. 8.0 The Basics of Effect Size Extraction and Statistical Applications for Meta-Analysis Robert M. Bernard Philip C. Abrami Concordia University

2. What is an Effect size? • A descriptive metric that characterizes the standardized difference (in SD units) between the mean of a control group and the mean of a treatment group (educational intervention) • Can also be calculated from correlational data derived from pre-experimental designs or from repeated measures designs

3. Characteristics of Effect Sizes • Can be positive or negative • Interpreted as a z-score, in SD units, although individual effect sizes are not part of a z-score distribution • Can be aggregated with other effect sizes and subjected to other statistical procedures such as ANOVA and multiple regression • Magnitude interpretation: ≤ 0.20 is a small effect size, 0.50 is a moderate effect size and ≥ 0.80 is a large effect size(Cohen, 1992)

4. Zero Effect Size ES = 0.00 Control Group Intervention Group Overlapping Distributions

5. Moderate Effect Size ES = 0.40 Control Group Treatment Group

6. ES = 0.85 Control Condition Intervention Condition

7. Large Effect Size ES = 0.85 Control Group Intervention Condition

8. Percentage Interpretation of Effect Sizes • ES = 0.00means that the average treatment participant outperformed0%of the control participants • ES = 0.40means that the average treatment participant outperformed65%of the control participants(from the Unit Normal Distribution) • ES = 0.85means that the average treatment participant outperformed80%of the control participants

9. Independence of Effect Sizes • Ideally, multiple effect sizes extracted from the same study should be independent from one another • This means that the same participants should not appear in more than one effect size • In studies with one control condition and multiple treatments, the treatments can be averaged, or one may be selected at random • Using effect sizes derived from different measures on the same participants is legitimate

10. Independence: Treatments & Measures One outcome R O1 R X1 O1 R X2 O1 R X3 O1 R O1 R Xpooled O1 Two outcomes, one for O1 and one for O2 R O1O2 R X1 O1O2

11. Effect Size Extraction • Effect size extraction is the process of identifying relevant statistical data in a study and calculating an effect size based on those data • All effect sizes should be extracted by two coders, working independently • Coders’ results should be compared and a measure of inter-coder agreement calculated and recorded • In cases of disagreement, coders should resolve the discrepancy in collaboration

12. ES Calculation: Descriptive Statistics

13. Examples from Three Studies

14. Extracting Effect Sizes in the Absence of Descriptive Statistics • Inferential Statistics (t-test, ANOVA, ANCOVA, etc.) when the exact statistics are provided • Levels of significance, such as p < .05, when the exact statistics are not given (t can be set at the conservative t = 1.96 (Glass, McGaw & Smith, 1981; Hedges, Shymansky & Woodworth, 1989) • Studies not reporting sample sizes for control and experimental groups should be considered for exclusion

15. Other Codable Data Regarding Effect size • Type of statistical data used to extract effect size (e.g., descriptives, t-value) • Type of effect size, such as posttest only, adjusted in ANCOVA, etc. • Direction of the statistical test • Reliability of dependent measure • In pretest/posttest design, the correlation between pretest and posttest

16. Examples from CT Meta-Analysis • Study 1: pretest/posttest, one-group design, all descriptives present • Study 2: posttest only, two-group design, all descriptives present • Study 3: pretest/posttest, two-group design, all descriptives present • Coding Sheet for 3 studies

17. Mean and Variability ES+ Variability Note:Results from Bernard, Abrami, Lou, et al. (2004) RER

18. Variability of Effect Size • The standard error of each effect size is estimated using the following equation: The average effect size (d+) is tested using the following equation:withN – 2degrees of freedom(Hedges & Olkin, 1985).

19. Testing Homogeneity of Effect Size Note the similarity to a t-ratio. Q is tested using the sampling distribution of 2 with k – 1 degrees of freedom where k is the number of effect sizes (Hedges & Olkin, 1985).

20. Homogeneity vs. Heterogeneity of Effect Size • If homogeneity of effect size is established, then the studies in the meta-analysis can be thought of as sharing the same effect size (i.e., the mean) • If homogeneity of effect size is violated (heterogeneity of effect size), then no single effect size is representative of the collection of studies (i.e., the “true” average effect size remains unknown)

21. Example with Fictitious Data *d+ is not significant, p > .05; **2 is significant, p < .05

22. Units of SD Favors Control Favors Treatment Graphing the Distribution of Effect Sizes Forrest Plot

23. Statistics in Comprehensive Meta-Analysis™ Note:Results from Bernard, Abrami, Lou, et al. (2004) RER Comprehensive Meta-Analysis 1.0 is a trademark of BioStat®

24. Examining Study Features • Purpose: to attempt to explain variability in effect size • Any nominally coded study feature can be investigated • In addition to mean effect size, variability should be investigated • Study features with small ks may be unstable

25. Females d+ = +0.24 k = 32 Males d+ = –0.14 k = 18 Examining the Study Feature Gender Overall Effect d+ = +0.14 k = 60

26. ANOVA on Levels of Study Features Note:Results from Bernard, Abrami, Lou, et al. (2004) RER

27. Sensitivity Analysis • Tests the robustness of the findings • Asks the question: Will these results stand up when potentially distorting or deceptive elements, such as outliers, are removed? • Particularly important to examine the robustness of the effect sizes of study features, as these are usually based on smaller numbers of outcomes

28. Meta-Regression • An adaptation of multiple linear regression • Effect sizes weighted by in regression • Used to model study features and blocks of study features with the intention of explaining variation in effect size • Standard errors , test statistics (z) and confidence intervals for individual predictors must be adjusted (Hedges & Olkin, 1984)

29. Selected References Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How Does Distance Education Compare to Classroom Instruction? A Meta-Analysis of the Empirical Literature. Review of Educational Research, 74(3), 379-439. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage. Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED 309 952].