1 / 22

Meta-Analysis and Strategy Research

Meta-Analysis and Strategy Research. Dan R. Dalton Kelley School of Business Indiana University. A [Very] Brief History of Research Synthesis. Averaging Correlations? Combining Significance Levels? The Narrative Review (aka “Counting” Review) Gene Glass (1976) “Invents” Meta-Analysis

louisa
Download Presentation

Meta-Analysis and Strategy Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Meta-Analysis and Strategy Research Dan R. Dalton Kelley School of Business Indiana University

  2. A [Very] Brief History of Research Synthesis • Averaging Correlations? • Combining Significance Levels? • The Narrative Review (aka “Counting” Review) • Gene Glass (1976) “Invents” Meta-Analysis • Early Critics – “An Exercise in Mega-Silliness”

  3. An Example of Meta-Analysis(Data Are Simulated) • Research Question: The Extent to which Equity Holdings by CEOs Are Related to Firms’ Financial Performance • Proposed Moderator: Expected that this Relationship Will be Moderated by the “Maturity of the Firm” (i.e., Firms that Are Five or Less Years Post-IPO vs. Other) • Studies Available for Meta-Analysis = 30 (10 are not significant, 10 are positive and significant, 10 are negative and significant)

  4. An Example of Meta-Analysis(Data Are Simulated) • R = Reliability • RR = Range Restriction • M = Moderator (1 = ≤ 5 yrs. Post-IPO; 2 = > 5 yrs. Post-IPO)

  5. “r” - A Bivariate Correlation • “r” vs. “d” • R-square • Deriving “r” from “d,” “t,” “F-score,” “Z,” “Chi-Square” … • “r” from Incomplete Information r = Z/sqrt n if “n” = 120 and Z = 1.96 with “r” unknown then r = +/ - .179 (i.e., 1.96/10.95)

  6. “r” – A Bivariate Correlation,cntd. • -17 to +17 and Enter What? • Discard the Study? • “r” and the Z-transformation? • “r” and Statistical Significance • And, a “Surprise” About Multiple Non-Significant Results

  7. “r” – A Bivariate Correlationand “n” • “r” As an Independent Variable, a Dependent Variable, a Control Variable, a Moderating Variable, a Mediating Variable… • “n” – The Sample Size from which the “r” Was Calculated • To Weight the Observed Correlation in Order to Calculate the Mean Weighted Correlation Across All of the Studies • “n” and the Correlation Matrix

  8. Ry (Reliability of y); Rx (Reliability of x) • Constructs vs. Observed Variables • Strategic Management Meta-Analyses with Ry = 1 and Rx = 1 • Strategic Management Variables Are Not That Good • The Choice of Ry and Rx Levels Is Counterintuitive – Lower Ry’s and Rx’s Will Improve the “Corrected r” • Ry and Rx at .8

  9. RRy & RRx (Range Restriction of y and x) • Analytical Issues of Range Restriction Have Become Increasingly Complex • In Strategic Management – RRy and RRx as Deliberate Selectivity in the Sample • Strategic Management and “Survival” Issues

  10. Moderation in Meta-Analysis • In Meta-Analysis a “Moderator” Is a Subgroup • Profligate Testing for Moderators • Capitalization on Chance • Loss of Statistical Power • Moderators Need Not Always Be Operationalized as a Dichotomy

  11. Meta-Analytic Procedures and Results

  12. Meta-Analytic Procedures and Results

  13. Meta-Analytic Results:Some Diagnostics • The Magnitude of the Mean True Score Correlation • Does the 90% Confidence Interval Include Zero? Suggests that the Mean True Score Is Not Significant • Does the 80% Credibility Interval (Difference between Low and High Estimates) Exceed .11? Suggests the Existence of a Moderator • Does the % Variance Attributable to Artifacts Exceed 75%? Suggests that a Moderator Is Unlikely • And, If the Tests Had Relied on Different Rx and Ry Values? [ .7 = .48; .8 = .417; .9 = .37 ]

  14. Results Summary • There is no simple relationship ( -.026, ns) between CEO equity holdings and firm financial performance. There is, however, some evidence of the existence of a moderating variable. • There is evidence of a moderating effect for time since IPO. The relationship between CEO equity holdings and firm financial performance for firms 5 years or less from IPO is .417, a significant relationship. The diagnostics suggest that a further moderating effect of this result is unlikely.

  15. Results Summary, cntd. • The relationship between CEO equity holdings and firm financial performance for firms more than 5 years from the IPO is -.144, a significant relationship. The diagnostics suggest that a further moderating effect of this result is likely.

  16. Other Issues in Meta-Analysis • Fixed vs. Random Effects Models • Random Effects Models – Population Parameters May Vary Across Studies • Fixed Effects Models – Population Parameters Are Invariant • “File Drawer” Problem • Unreported Null Results • “Fail Safe” Approach • The Issue Is Less a Matter of Fail Safe Algorithms than of Reliance on Too Few Studies

  17. Other Issues in Meta-Analysis, cntd. • Quality of Data • Outliers • Statistical Outliers • Entry Error Outliers • Sensitivity to Outliers • The General Question of Discarding Data • Disclosure and Replicability

  18. Other Issues in Meta-Analysis, cntd. • The Independence of Data • Entering Data that Are Clearly Not Independent • A Random Selection, Pooling, a Weighted “r”, a Weighted “n” • An Interesting Catch-22 • “Clearly Reflect the Same Construct” • Independence of Samples • Constructive Replication

  19. General Guidelines for Meta-Analysis • There is no need to transform the input values of “r”s. • When it is necessary to impute the value of “r,” set “r” = 0. • For observed variables, rely on .8 for the reliability of the dependent and independent variables. • With observed variables, it will rarely be necessary to assign a range restriction score.

  20. General Guidelines for Meta-Analysis, cntd. • Use a conservative 90% confidence interval for the meta-analysis diagnostics (for these data, 95% would be an interval of -.128 to .075, much wider than the -.112 to .059 reported). • Use a conservative 80% credibility interval for the meta-analysis diagnostics (for these data, the 90% would have been an interval of -.493 to .448, much wider than the -.389 to .336 reported). • Where the meta-analysis software provides an option, rely on a “Random Effects Model.”

  21. General Guidelines for Meta-Analysis, cntd. • Assuming every effort has been made for an exhaustive search for meta-analysis input data, you need not be concerned about “file drawer” issues • Neither weight nor exclude data on the basis of the quality of the study. Instead, run two meta-analyses and compare the results for the entire data set and a reduced data set without the troublesome data

  22. General Guidelines for Meta-Analysis, cntd. • Only under extremely rare conditions would there be any concerns about the independence of the data; accordingly, there is no need to combine data from separate “r”s in any manner. • No need to exclude outliers. Instead, run two meta-analyses and compare the results for the entire data set and a data set without the outliers.

More Related