Meta Analysis

1. Meta Analysis Sunday, September 17, 2006

2. Definition • Meta-analysis is a set of statistical procedures designed to accumulate experimental, quasi-experimental, and correlational results across independent studies that address a set of related research questions. • Unlike traditional research methods, meta-analysis uses the summary statistics from individual studies as the data points. • A key assumption of this analysis is that each study provides a differing estimate of the underlying relationship within the population. • By accumulating results across studies, researchers gain a more accurate representation of the population compared to individual study estimators.

3. Definition • Glass and colleagues (e.g., Glass, 1976; 1977; Glass &;Smith, 1977; McGraw &;Glass, 1980; Smith &;Glass, 1977; and Smith, Glass &;Miller, 1980) coined the term meta-analysis, and introduced most of the currently used procedures used in research today. • Meta-analysis refers to the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. • It requires a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature. • (Glass, 1976, p 3).

4. Why Use Meta-Analysis • Meta analysis can be used as a guide to answer the question. • Is the Research we are doing in Literacy making a difference for Blind & Visually Impaired people? • Meta-analysis helps to provide a systematic overview of quantitative research which examines a particular research question. • The appeal of meta analysis is that it combines all the research on one topic into one large study with many participants. • The danger is that in amalgamating a large set of different studies the construct definitions can become imprecise and the meaning of results can be difficult to interpret. • The first meta-analysis was performed by Karl Pearson in 1904, in an attempt to overcome the problem of reduced statistical power in studies with small sample sizes. • So, analyzing the results from a group of studies can allow for more accurate data analysis.

5. Effect Size • Meta analysis reports findings in terms of effect sizes.  • Effect size provides information about how much change is evident across all studies and for subsets of studies. • Effect Size allows for an apples-to-apples comparison & analysis of multiple studies. • Definition: “The degree to which a phenomenon is present in the population being studied.” • There are many different types of effect size, but they all fall into two main types: • Standardized mean difference (e.g., Cohen's d or Hedges g) • Correlation (e.g., Pearson's r) ,

6. Effect Size • The standardized mean effect size is basically computed as the difference score divided by the standard deviation of the scores. • In meta-analysis, effect sizes should also be reported with: • the number of studies and the number of effects used to create the estimate. • confidence intervals to help readers determine the consistency and reliability of the mean estimated effect size.

7. Effect Size Cont… • Because the results from different studies investigating different independent variables are measured on different scales, the dependent variable in a meta-analysis is some standardized measure of effect size. • To describe the results of comparative experiments the usual effect size indicator is the standardized mean difference (d) which is the standard score equivalent to the difference between means, or an odds ratio if the outcome of the experiments is a dichotomous variable (success versus failure). • A meta-analysis can be performed on studies that describe their findings in correlation coefficients. In these cases, the correlation itself is the indicator of the effect size. • For example, a meta-analysis could be performed on a collection of studies each of which attempts to estimate the incidence of left-handedness in various groups of people.

8. Process • What is your research question? What are you trying to achieve? • Set Criteria for the literature • Article from 1960 to 2005 only. • Each article must have an intervention & a TRUE control group. • Literature Review • Key word searches • Standards • Setting standards determines the evidence of causal validity. Evidence based research screening. (E.g., Experimental, Quasi-Experimental, or no randomization or true control group). • Reporting System • Provide key findings on each study. • Briefly describe the topic and the intervention used. • Team Review • Each article should be reviewed by 3 people for validity.

9. References • Hattie, J. (1992). Self-concept. NJ: Lawrence Erlbaum. • Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment. American Psychologist, 48, 1181-1201. • Smith, M. L., Glass, G. V., & Miller, T. I. (1980).  The benefits of psychotherapy. Baltimore: Johns Hopkins University Press. • Bushman, B. J., & Wells, G. L. (2001). Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Personality and Social Psychology Bulletin, 27, 1123-1130. • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Academic Press. • Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8. • Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379. • Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage.