1 / 29

The Basics of Effect Size Extraction and Statistical Applications for Meta-Analysis

8.0. The Basics of Effect Size Extraction and Statistical Applications for Meta-Analysis. Robert M. Bernard Philip C. Abrami Concordia University. What is an Effect size?.

ewa
Download Presentation

The Basics of Effect Size Extraction and Statistical Applications for Meta-Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. 8.0 The Basics of Effect Size Extraction and Statistical Applications for Meta-Analysis Robert M. Bernard Philip C. Abrami Concordia University

  2. What is an Effect size? • A descriptive metric that characterizes the standardized difference (in SD units) between the mean of a control group and the mean of a treatment group (educational intervention) • Can also be calculated from correlational data derived from pre-experimental designs or from repeated measures designs

  3. Characteristics of Effect Sizes • Can be positive or negative • Interpreted as a z-score, in SD units, although individual effect sizes are not part of a z-score distribution • Can be aggregated with other effect sizes and subjected to other statistical procedures such as ANOVA and multiple regression • Magnitude interpretation: ≤ 0.20 is a small effect size, 0.50 is a moderate effect size and ≥ 0.80 is a large effect size(Cohen, 1992)

  4. Zero Effect Size ES = 0.00 Control Group Intervention Group Overlapping Distributions

  5. Moderate Effect Size ES = 0.40 Control Group Treatment Group

  6. ES = 0.85 Control Condition Intervention Condition

  7. Large Effect Size ES = 0.85 Control Group Intervention Condition

  8. Percentage Interpretation of Effect Sizes • ES = 0.00means that the average treatment participant outperformed0%of the control participants • ES = 0.40means that the average treatment participant outperformed65%of the control participants(from the Unit Normal Distribution) • ES = 0.85means that the average treatment participant outperformed80%of the control participants

  9. Independence of Effect Sizes • Ideally, multiple effect sizes extracted from the same study should be independent from one another • This means that the same participants should not appear in more than one effect size • In studies with one control condition and multiple treatments, the treatments can be averaged, or one may be selected at random • Using effect sizes derived from different measures on the same participants is legitimate

  10. Independence: Treatments & Measures One outcome R O1 R X1 O1 R X2 O1 R X3 O1 R O1 R Xpooled O1 Two outcomes, one for O1 and one for O2 R O1O2 R X1 O1O2

  11. Effect Size Extraction • Effect size extraction is the process of identifying relevant statistical data in a study and calculating an effect size based on those data • All effect sizes should be extracted by two coders, working independently • Coders’ results should be compared and a measure of inter-coder agreement calculated and recorded • In cases of disagreement, coders should resolve the discrepancy in collaboration

  12. ES Calculation: Descriptive Statistics

  13. Examples from Three Studies

  14. Extracting Effect Sizes in the Absence of Descriptive Statistics • Inferential Statistics (t-test, ANOVA, ANCOVA, etc.) when the exact statistics are provided • Levels of significance, such as p < .05, when the exact statistics are not given (t can be set at the conservative t = 1.96 (Glass, McGaw & Smith, 1981; Hedges, Shymansky & Woodworth, 1989) • Studies not reporting sample sizes for control and experimental groups should be considered for exclusion

  15. Other Codable Data Regarding Effect size • Type of statistical data used to extract effect size (e.g., descriptives, t-value) • Type of effect size, such as posttest only, adjusted in ANCOVA, etc. • Direction of the statistical test • Reliability of dependent measure • In pretest/posttest design, the correlation between pretest and posttest

  16. Examples from CT Meta-Analysis • Study 1: pretest/posttest, one-group design, all descriptives present • Study 2: posttest only, two-group design, all descriptives present • Study 3: pretest/posttest, two-group design, all descriptives present • Coding Sheet for 3 studies

  17. Mean and Variability ES+ Variability Note:Results from Bernard, Abrami, Lou, et al. (2004) RER

  18. Variability of Effect Size • The standard error of each effect size is estimated using the following equation: The average effect size (d+) is tested using the following equation:withN – 2degrees of freedom(Hedges & Olkin, 1985).

  19. Testing Homogeneity of Effect Size Note the similarity to a t-ratio. Q is tested using the sampling distribution of 2 with k – 1 degrees of freedom where k is the number of effect sizes (Hedges & Olkin, 1985).

  20. Homogeneity vs. Heterogeneity of Effect Size • If homogeneity of effect size is established, then the studies in the meta-analysis can be thought of as sharing the same effect size (i.e., the mean) • If homogeneity of effect size is violated (heterogeneity of effect size), then no single effect size is representative of the collection of studies (i.e., the “true” average effect size remains unknown)

  21. Example with Fictitious Data *d+ is not significant, p > .05; **2 is significant, p < .05

  22. Units of SD Favors Control Favors Treatment Graphing the Distribution of Effect Sizes Forrest Plot

  23. Statistics in Comprehensive Meta-Analysis™ Note:Results from Bernard, Abrami, Lou, et al. (2004) RER Comprehensive Meta-Analysis 1.0 is a trademark of BioStat®

  24. Examining Study Features • Purpose: to attempt to explain variability in effect size • Any nominally coded study feature can be investigated • In addition to mean effect size, variability should be investigated • Study features with small ks may be unstable

  25. Females d+ = +0.24 k = 32 Males d+ = –0.14 k = 18 Examining the Study Feature Gender Overall Effect d+ = +0.14 k = 60

  26. ANOVA on Levels of Study Features Note:Results from Bernard, Abrami, Lou, et al. (2004) RER

  27. Sensitivity Analysis • Tests the robustness of the findings • Asks the question: Will these results stand up when potentially distorting or deceptive elements, such as outliers, are removed? • Particularly important to examine the robustness of the effect sizes of study features, as these are usually based on smaller numbers of outcomes

  28. Meta-Regression • An adaptation of multiple linear regression • Effect sizes weighted by in regression • Used to model study features and blocks of study features with the intention of explaining variation in effect size • Standard errors , test statistics (z) and confidence intervals for individual predictors must be adjusted (Hedges & Olkin, 1984)

  29. Selected References Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How Does Distance Education Compare to Classroom Instruction? A Meta-Analysis of the Empirical Literature. Review of Educational Research, 74(3), 379-439. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage. Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED 309 952].

More Related