1 / 74

Meta-analysis

Sessions 1.2-1.3: Effect Size Calculation. Funded through the ESRC’s Researcher Development Initiative. Meta-analysis. Department of Education, University of Oxford. Sessions 1.2-1.3: Effect Size Calculation. 2. Effect size calculation. The effect size makes meta-analysis possible

jude
Download Presentation

Meta-analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sessions 1.2-1.3: Effect Size Calculation Funded through the ESRC’s Researcher Development Initiative Meta-analysis Department of Education, University of Oxford

  2. Sessions 1.2-1.3: Effect Size Calculation 2

  3. Effect size calculation • The effect size makes meta-analysis possible • It is based on the “dependent variable” (i.e., the outcome) • It standardizes findings across studies such that they can be directly compared • Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must • be comparable across studies (standardization) • represent magnitude & direction of the relationship • be independent of sample size • Different studies in same meta-analysis can be based on different statistics, but have to transform each to a standardized effect size that is comparable across different studies

  4. Sample size, significance and d effect size XLS

  5. Sample size, significance and d effect size XLS ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 5 5

  6. Simulate ds on homemade calculator(ES.xls) • Change direction of effects • Change Ns (equal or same?) • Change SDs XLS ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 6 6

  7. = Control = Treatment Effect size as proportion in the Treatment group doing better than the average Control group person 69% of T above 79% of T above 57% of T above 7

  8. = Control = Treatment Effect size as proportion of success in the Treatment versus Control group (Binomial Effect Size Display = BESD): Success: 55% of T, 45% of C Success: 62% of T, 38% of C Success: 68% of T, 32% of C 8

  9. Long focus on significance level (safe-guarding against Type I (a) error) – today focus on practical and meaningful significance. Cohen, J. (1994). The earth is round (p < .05), American Psychologist, 49, 997–1003. Why effect size? 9

  10. A short history of the effect size (Huberty, 2002; see also Olejnik & Algina, 2000 for review of effect sizes) 10

  11. Power: “Finding what is out there” Type II (b) error “not finding what is out there” Power (1 – b): the probability of rejecting a false H0 hypothesis Power of .80 or .90 in primary research Power and effect size 11

  12. Power, sought effect size, at significance level a = .05 in primary research (prior to conducting study) 12

  13. How meaningful is a “small” effect size? • A small effect size changed the course of an RCT in 1987: placebo group participants were given aspirin instead (see Rosenthal, 1994, p. 242) XLS 13

  14. Effect sizes Within the one meta-analysis, can include studies based on any combination of statistical analysis (e.g., t-tests, ANOVA, correlation, odds-ratio, chi-square, etc). The “art” of meta-analysis is how to compute effect sizes based on non-standard designs and studies that do not supply complete data (see Lipsey&Wilson_AppB.pdf). Convert all effect sizes into a common metric based on the “natural” metric given research in the area. E.g. d, r, OR ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 14

  15. Effect size calculation Standardized mean difference Group contrast research Treatment groups Naturally occurring groups Inherently continuous construct Correlation coefficient Association between inherently continuous constructs Odds-ratio Group contrast research Treatment or naturally occurring groups Inherently dichotomous construct Regression coefficients and other multivariate effects Requires access to covariance-variance (correlation) matrices for each included study 15

  16. Calculating ds (1) Means and standard deviations Almost all test statistics can be transformed into an standardized effect size “d” Correlations d P-values F-statistics t-statistics “other” test statistics 16 ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 16

  17. Represents a standardized group contrast on an inherently continuous measure Uses the pooled standard deviation Commonly called “d” Calculating ds (1) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)

  18. Cohen’s d Hedge’s g Glass’s D Various contrast effect sizes ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 18

  19. Calculating d (1) using Ms, SDs and ns Remember to code treatment effect in positive direction! ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 19

  20. ES_calculator.xls 20 20

  21. Calculating d (2) using ES calculator, using Ms, ns, and t-value 21

  22. Calculating d (3) using ES calculator, using ns, and t-value • The treatment group scored higher than the control group at Time 2 (t[28]= 4.11; p<.001). • From sample description we learn that n1 = n2 22 22

  23. Hedges proposed a correction for small sample size bias (ns < 20) Must be applied before analysis Calculating d (3) correcting for small sample bias 23

  24. Calculating d (4) using ES calculator, using ns, and F-value Remember: in a two-group ANOVAF = t2 24 24

  25. Calculating d (5) using ES calculator, using p-value “The mean-level comparison was not significant (p = .53)” 25 25

  26. T-test table df = (n1 + ns –2) Sometimes authors only report e.g., p<.01 (n = 22). If so, use a conservative approach to reading the t-test table. NOTE: When p = n.s. some researchers code d = 0 in data base 26 26

  27. Example dataset so far (1) (ES_enter.sav): 27 27

  28. Use all available tools for calculating the following 5 effect sizes • ES 6: MT = 21, MC = 20, nT = 60, nC = 60, t = .55 • ES 7: MT = 103.5, MC = 100, SDT = 22.0, SDC = 18.5, nT = 45, nC = 35, • ES 8: nT = 45, nC = 40, p <.05 • ES 9: nT = 100, nC = 120, F = 8.73 • ES 10: nT = 200, nC = 160, t = 5.66 (see electronic document: “Correct ds for 5 effect sizes.doc”) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 28

  29. Example dataset so far (2) (ES_enter.sav): 29 29

  30. Calculating d (11) using ES calculator, using number of successful outcomes per group 30 30

  31. Calculating d (11) using ES calculator, using number of successful outcomes per group 31 31

  32. Calculating d (12) using ES calculator, using proportion of successes per group (53% vs. 48.5%) 32 32

  33. Calculating d (13) using paired t-test (only one experimental group; “each person their own control”) Don’t use the SD of the change score! r = correlation between Time 1 and Time 2 33 33

  34. Calculating d (14) using paired t-test (only one experimental group) • n (pairs) = 90, t-value = 6.5, r = .70 34 34

  35. Calculating d (15) • “The 20 participants increased .84 z-scores between time 1 and time 2 (p<.01)” • ES = .84 • Correct for small sample bias 35 35

  36. Example dataset so far 3 (ES_enter.sav): Method difference: mean contrast and gain scores 36 36

  37. Summary of equations from Lipsey & Wilson (2001) (for more formulae see Lipsey & Wilson Appendix B) 37 37

  38. The effect sizes are weighted by the inverse of the variance to give more weight to effects based on larger sample sizes Variance for mean level comparison is calculated as The standard error of each effect size is given by the square root of the sampling variance SE =  vi Weighting for mean-level differences ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 38 38

  39. Enter_w.xls ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 39 39

  40. SE for gain scores Inverse variance for gain scores Weighting for gain scores T1 and T2 scores are dependent so we need to get correlation between T1 and T2 into equation (not always reported) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 40 40

  41. Enter_w.xls XLS ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 41 41

  42. Compute the weighted mean ES and s.e. of the ES in SPSS (var_ofES.sps) (1) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 42 42

  43. Compute the weighted mean ES and s.e. of the ES in SPSS (var_ofES.sps) (2) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 43 43

  44. Weight the ES by the inverse of the s.e. The average ES Standard error of the ES Compute the weighted mean ES and s.e. of the ES ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 44 44

  45. Enter_w.xls ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 45 45

  46. ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 46 46

  47. Does average of ES converge toward the average of the largest (n) study? Funnel plot for x = sample size, y = ES 95% C.I. = ±1.96 * s.e. 99% C.I. = ±2.58 * s.e. 99.9% C.I. = ±3.29 * s.e. 47

  48. ES in smaller sample has larger standard error (s.e.) Funnel plot including s.e. of ES 48

  49. Population and sample Population N = ‘size’ m = ‘mean’ d = ‘effect size’ Sample n = ‘size’ m = ‘mean’ d = ‘effect size’ Interval estimates • The “likely” population parameter is the sample parameter ± uncertainty • Standard errors (s.e.) • Confidence intervals (C.I.) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 49 49

  50. Calculating rs Means and standard deviations (d) Almost all test statistics can be transformed into an standardized effect size “r” c2f r P-values F-statistics t-statistics “other” test statistics ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 50

More Related