What is Meta-analysis?

1. NCRM Research Methods Festival University of Oxford What is Meta-analysis? Dept of Education, University of Oxford

2. Primary versus secondary data analysis Traditionally, social science researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue: 1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation. 2. Systematic review-- systematic, qualitative review of published research in a particular field 3. Secondary Data Analyses -- using large (typically public) databases 4. Re-analyses of published studies -- often in ways critical of the original study.

3. What is meta-analysis? • Systematic synthesis of various studies on a particular research question Do boys or girls have higher self-concepts? • Collect all studies relevant to a topic Find all published journal articles on the topic • An effect size (the ‘dependent variable’) is calculated for each outcome Determine the size/direction of gender difference for each study • “Content analysis” code characteristics of the study; age, setting, ethnicity, self-concept domain (math, physical, social), etc. • Effect sizes with similar features are grouped together and compared; tests moderator variables Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc. 3

4. A blend of qualitative and quantitative approaches • Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative) • Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative) • Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative)

5. Benefits of meta-analysis • Compared to traditional literature reviews: • (1) there is a definite methodology employed in the research analysis; and  • (2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis. • Therefore less biased and more replicable

6. Benefits of meta-analysis • Increased power: increases the chance of detecting a true treatment effect • Improved precision: with more information than a single study, the treatment effect estimate is improved • When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies • Easy to interpret summary statistics (useful if communicating findings to a non-academic audience)

7. Why is meta-analysis important? Generalisability • The essence of good science is replicable and generalisable results. • Do we get the same answer to important research questions when we run the study again? • The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question. • Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency?

8. Why is meta-analysis important? • Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines • Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it • Relied upon by policymakers • Important that we understand the method, whether we conduct or consume meta-analytic research • Should be one of the topics covered in all introductory research methodology courses

9. When is meta-analysis appropriate? • There exists a critical mass of comparable studies designed to address a common research question. • Data are presented in a form that allows the meta-analyst to compute an effect size for each study. • Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study.

10. Schulze, R. (2007). The state and the art of meta-analysis . ZeitschriftfürPsychologie/Journal of Psychology, 215, 87-89. The number of meta-analyses is increasing at a rapid rate.

11. Where are meta-analyses done?

12. Psychology: Citations Psychology: Articles

13. Psychology: Where it all began • Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis . Psychological Bulletin, 110, 26-46.Times Cited: 471 • Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis . Child Development, 56, 1479-1498.Times Cited: 570 • Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis . Psychological Bulletin, 89, 47-62.Times Cited: 426 • Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review . Personnel Psychology, 44, 703-742Times Cited: 387 • Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis . Psychological Bulletin, 104, 53-69.Times Cited: 316 • Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis . Psychological Bulletin, 97, 251-273.Times Cited: 263.

14. Education: Widely Cited Meta-analyses • De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment . Child Development, 68, 571-591.Times Cited: 340 • Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development: The truth about false belief . Child Development, 72, 655-684.Times Cited: 276 • Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups . Review of Educational Research, 64, 1-35. Times Cited: 235 • Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990 . Health Education Research, 7, 403-430.Times Cited: 207 • Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer-Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, 525-544.Times Cited: 198.

15. Business/Management: Widely Cited Meta-analyses • Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research . Journal of Consumer Research, 15, 325-343.Times Cited: 515 • Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings . Organizational Behavior and Human Decision Processes, 36, 16-78.Times Cited: 401 • Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings . IEEE Transactions On Engineering Management, 29, 28-4. Times Cited: 269. • Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7, 385-425. Times Cited: 203. • Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis . Journal Of Marketing Research, 22, 103-118. Times Cited: 189.

16. Most Widely Cited Meta-analyses are in Medicine • Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1-12. Times Cited:2008 • Boushey Cj, Beresford Saa, Omenn Gs, Et . Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, 1049-1057. Times Cited: 2,128 • Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, 899-909. Times Cited:1,591 • Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, 1-29. Times Cited: 1,422

17. Classic Meta-analysis:Smith Ml, Glass GV (1977): Meta-analysis Of Psychotherapy Outcome Studies. American Psychologist, 32, 752-760. Times Cited: 840. Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant. Results of 375 controlled evaluations of psychotherapy and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy. On the average, the typical therapy client is better off than 75% of untreated individuals. Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral). ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 17

18. Steps in a meta-analysis

19. Defining a population of studies and finding publications • Need to have explicit inclusion and exclusion criteria • The broader the research domain, the more detailed they tend to become • Refine criteria as you interact with the literature • Components of a detailed criteria • distinguishing features • research respondents • key variables • research methods • cultural and linguistic range • time frame • publication types

20. Locate and collate studies • Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC) • Examine the reference lists of included studies to find other relevant studies • If including unpublished data, email researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International

21. Pilot coding • Random selection of papers coded by both coders • Meet to compare code sheets • Where there is discrepancy, discuss to reach agreement • Amend code materials/definitions in code book if necessary • May need to do several rounds of piloting, each time using different papers

22. Develop code materials Code Sheet Code Book/manual 1 99 2 1 87 41 46 Publication type (1-5) Journal article Book/book chapter Thesis or doctoral dissertation Technical report Conference paper __ Study ID _ _ Year of publication __ Publication type (1-5) __ Geographical region (1-7) _ _ _ _ Total sample size _ _ _ Total number of males _ _ _ Total number of females ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)

23. Effect size calculation • The effect size makes meta-analysis possible • It is the “dependent variable” • It standardizes findings across studies such that they can be directly compared • Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must • be comparable across studies (generally requires standardization) • represent the magnitude and direction of the relationship of interest • be independent of sample size

24. Effect size calculation In an intervention study with experimental and control groups, the effect size might be: In a gender difference study, the effect size might be: Represents a standardized group contrast on an inherently continuous measure Uses the pooled standard deviation (some situations use control group standard deviation) Commonly called “d” 24

25. Effect size calculation Means and standard deviations Almost all test statistics can be transformed into an standardized effect size “d” Correlations d P-values F-statistics t-statistics Lipsey & Wilson (2001) present formulae for calculating effect sizes from different information “other” test statistics ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)

26. Sample sizes Each study is one line in the data base Variance of the effect size Effect size Duration Reliability of the instrument Constructing a database

27. Analytical methods • There are various ways of analysing meta-analytic data • Three main methods based on different statistical assumptions: • Fixed effects models • Random effects models • Multilevel models • These will be discussed in the afternoon workshop

28. Conclusion • Meta-analysis is a method for synthesising and analysing the research literature on a particular topic • The essence of good science is replicable and generalisable results. • Increasingly sophisticated • Continuously evolving • For more information about the meta-analysis training courses that we offer, please see http://education.ox.ac.uk/research/resgroup/self/training.php