Best Meta-Analysis Practice

1. Best Meta-Analysis Practice As of Spring 2015

2. Before you start… • Define precisely the research question • Know how you will analyze the data before you collect them • Are there sufficient studies for • a reasonable CI for an overall effect? • power for tests of moderators? • Do you have time and resources to search, gather and code data?

3. Data Collection & Analysis 1 • Multiple coders for all or at least some of the studies – analyze agreement/reliability • Exhaustive search for grey and unpublished studies • Appropriate model(s) for data analysis • Random effects • Nesting or dependencies • Moderator analyses • Bias & Sensitivity analyses

4. Data Collection & Analysis 2 • Coding studies • Choice of effect size • Assumptions (e.g., if only p value reported) for derivation • Transformations from one effect size to another • Code-book description • Consensus method • Description of raters • Rater training, N of raters for each judgment • Rater Agreement/reliability • Alpha/intraclass correlation • Kappa • Percent agreement • Test-retest

5. Publication 1 • Title – MA? Synthesis? • Abstract • Include numbers • Introduction • Clear statement of research questions • Controversy in lit? • Confirm/refute popular opinion • Create powerful test, e.g., differential validity by race • Probs with previous metas, e.g., average all kinds of teams • Define constructs • Hypothesize moderators

6. Publication 2 • Data Collection, Sources • Conference proceedings/abstracts • Dissertations • Electronic databases searched; include terms, keywords • Google scholar • Hand searches of literature – a must include because you get stuff you don’t get in electronic databases • Listservs • Other sources of documents (e.g., key researchers/informants) • Contact authors directly • Librarian (hire if you can afford) • Reference sections of articles • Unpublished manuscripts • Websites of authors in the area

7. Publication 3 • What data must you include (your stated analysis requires what)? • Effect size • Moderator • Sample size • Specific means, standard deviations, etc.

8. Publication 4 • Eligibility, Inclusion/Exclusion • Country • Language • Measures allowed • Characteristics of the measures (e.g., minimum number of items, reliability, etc.) • E.g., NEO vs. big five generic • Specific measures (e.g,. facet vs. general satisfaction) • Participants (e.g., all male) • Publications (type included) • Journal (e.g., APA journal) • Book vs. article, etc. • Published vs. not vs. subsequently published • Study design (e.g., specific control groups, repeated measures, etc.) • Time frame

9. Publication 5 • Report coding efforts – replicable? Reliable? • Report Summary Effects (usually mean) • Model and weights used • Anti-bias methods (e.g., d to g) • Overall mean, CI and PI (k and N desirable) • REVC and CI (k and N desirable) • Sensitivity analyses • Outlier • Publication • Other

10. Publication 6 • Report Moderator Tests • Model: categorical, continuous, both • Significance test result • Practical test result (show magnitude of effect: subgroup means, regression coefficients) • Reduction in REVC, R-square analog • Power of test? • Any moderator tests not hypothesized in advance should be clearly indicated

11. Publication 7 • Graphs • Flow chart – search thru analysis (desirable) • Forest plot – overall summary • Possible additional graphs • Boxplot or stem-leaf if too many ES • Funnel plot (contour ehanced); trim & fill – publication bias • Cumulative forest – publication bias • Omit-one forest – outliers

12. Publication 8 • Discussion • Conclusion with respect to initial research question; results clearly linked to questions • Meaning of results in terms of study (e.g., Binomial Effect Size Display) • Future research e.g., other moderators, what are gaps based on what is known • Discussion of credible threats / study limits

13. Publication 8 • Discussion • Conclusion with respect to initial research question; results clearly linked to questions • Meaning of results in terms of study (e.g., Binomial Effect Size Display) • Future research e.g., other moderators, what are gaps based on what is known • Discussion of credible threats / study limits