What is Meta-analysis

1. What is Meta-analysis? NCRM Research Methods Festival University of Oxford

2. Primary versus secondary data analysis Traditionally, social science researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue: 1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation. 2. Systematic review -- systematic, qualitative review of published research in a particular field 3. Secondary Data Analyses -- using large (typically public) databases 4. Re-analyses of published studies -- often in ways critical of the original study. 2 ��

3. What is meta-analysis? Systematic synthesis of various studies on a particular research question Do boys or girls have higher self-concepts? Collect all studies relevant to a topic Find all published journal articles on the topic An effect size (the �dependent variable�) is calculated for each outcome Determine the size/direction of gender difference for each study �Content analysis� code characteristics of the study; age, setting, ethnicity, self-concept domain (math, physical, social), etc. Effect sizes with similar features are grouped together and compared; tests moderator variables Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc.

4. A blend of qualitative and quantitative approaches Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative) Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative) Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative) 4

5. Benefits of meta-analysis Compared to traditional literature reviews: (1) there is a definite methodology employed in the research analysis; and� (2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis. Therefore less biased and more replicable 5

6. Benefits of meta-analysis Increased power: increases the chance of detecting a true treatment effect Improved precision: with more information than a single study, the treatment effect estimate is improved When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies Easy to interpret summary statistics (useful if communicating findings to a non-academic audience) 6

7. The essence of good science is replicable and generalisable results. Do we get the same answer to important research questions when we run the study again? The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question. Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency? 7 Why is meta-analysis important? Generalisability

8. Why is meta-analysis important? Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it Relied upon by policymakers Important that we understand the method, whether we conduct or consume meta-analytic research Should be one of the topics covered in all introductory research methodology courses 8

9. When is meta-analysis appropriate? There exists a critical mass of comparable studies designed to address a common research question. Data are presented in a form that allows the meta-analyst to compute an effect size for each study. Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study. 9

10. Schulze, R. (2007). The state and the art of meta-analysis . Zeitschrift f�r Psychologie/Journal of Psychology, 215, 87-89. 10

11. Where are meta-analyses done? 11

12. 12

13. Psychology: Where it all began Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis . Psychological Bulletin, 110, 26-46. Times Cited: 471 Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis . Child Development, 56, 1479-1498. Times Cited: 570 Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis . Psychological Bulletin, 89, 47-62. Times Cited: 426 Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review . Personnel Psychology, 44, 703-742 Times Cited: 387 Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis . Psychological Bulletin, 104, 53-69. Times Cited: 316 Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis . Psychological Bulletin, 97, 251-273. Times Cited: 263. 13

14. Education: Widely Cited Meta-analyses De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment . Child Development, 68, 571-591. Times Cited: 340 Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development: The truth about false belief . Child Development, 72, 655-684. Times Cited: 276 Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups . Review of Educational Research, 64, 1-35. Times Cited: 235 Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990 . Health Education Research, 7, 403-430. Times Cited: 207 Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer-Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, 525-544. Times Cited: 198. 14

15. Business/Management: Widely Cited Meta-analyses Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research . Journal of Consumer Research, 15, 325-343. Times Cited: 515 Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings . Organizational Behavior and Human Decision Processes, 36, 16-78. Times Cited: 401 Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings . IEEE Transactions On Engineering Management, 29, 28-4. Times Cited: 269. Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7, 385-425. Times Cited: 203. Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis . Journal Of Marketing Research, 22, 103-118. Times Cited: 189. 15

16. Most Widely Cited Meta-analyses are in Medicine Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1-12. Times Cited:2008 Boushey Cj, Beresford Saa, Omenn Gs, Et . Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, 1049-1057. Times Cited: 2,128 Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, 899-909. Times Cited:1,591 Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, 1-29. Times Cited: 1,422 16

17. Classic Meta-analysis: Smith Ml, Glass GV (1977): Meta-analysis Of Psychotherapy Outcome Studies. American Psychologist, 32, 752-760. Times Cited: 840.  Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant. Results of 375 controlled evaluations of psychotherapy and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy. On the average, the typical therapy client is better off than 75% of untreated individuals. Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral). 17

18. Steps in a meta-analysis 18

19. Defining a population of studies and finding publications Need to have explicit inclusion and exclusion criteria The broader the research domain, the more detailed they tend to become Refine criteria as you interact with the literature Components of a detailed criteria distinguishing features research respondents key variables research methods cultural and linguistic range time frame publication types 19

20. Locate and collate studies Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC) Examine the reference lists of included studies to find other relevant studies If including unpublished data, email researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International 20

21. Pilot coding Random selection of papers coded by both coders Meet to compare code sheets Where there is discrepancy, discuss to reach agreement Amend code materials/definitions in code book if necessary May need to do several rounds of piloting, each time using different papers 21

22. Develop code materials __ Study ID _ _ Year of publication __ Publication type (1-5) __ Geographical region (1-7) _ _ _ _ Total sample size _ _ _ Total number of males _ _ _ Total number of females 22

23. Effect size calculation The effect size makes meta-analysis possible It is the �dependent variable� It standardizes findings across studies such that they can be directly compared Any standardized index can be an �effect size� (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must be comparable across studies (generally requires standardization) represent the magnitude and direction of the relationship of interest be independent of sample size 23

24. Effect size calculation Represents a standardized group contrast on an inherently continuous measure Uses the pooled standard deviation (some situations use control group standard deviation) Commonly called �d�

25. Effect size calculation 25 Within the one meta-analysis, can include studies based on any combination of statistical analysis (e.g., t-tests, ANOVA, multiple regression, correlation, odds-ratio, chi-square, etc). However, you have to convert each of these to a common �effect size� metric. Within the one meta-analysis, can include studies based on any combination of statistical analysis (e.g., t-tests, ANOVA, multiple regression, correlation, odds-ratio, chi-square, etc). However, you have to convert each of these to a common �effect size� metric.

26. Constructing a database 26

27. Analytical methods There are various ways of analysing meta-analytic data Three main methods based on different statistical assumptions: Fixed effects models Random effects models Multilevel models These will be discussed in the afternoon workshop 27

28. Conclusion Meta-analysis is a method for synthesising and analysing the research literature on a particular topic The essence of good science is replicable and generalisable results. Increasingly sophisticated Continuously evolving For more information about the meta-analysis training courses that we offer, please see http://education.ox.ac.uk/research/resgroup/self/training.php 28