Meta-Analysis and Strategy Research

1. Meta-Analysis and Strategy Research Dan R. Dalton Kelley School of Business Indiana University

2. A [Very] Brief History of Research Synthesis • Averaging Correlations? • Combining Significance Levels? • The Narrative Review (aka “Counting” Review) • Gene Glass (1976) “Invents” Meta-Analysis • Early Critics – “An Exercise in Mega-Silliness”

3. An Example of Meta-Analysis(Data Are Simulated) • Research Question: The Extent to which Equity Holdings by CEOs Are Related to Firms’ Financial Performance • Proposed Moderator: Expected that this Relationship Will be Moderated by the “Maturity of the Firm” (i.e., Firms that Are Five or Less Years Post-IPO vs. Other) • Studies Available for Meta-Analysis = 30 (10 are not significant, 10 are positive and significant, 10 are negative and significant)

4. An Example of Meta-Analysis(Data Are Simulated) • R = Reliability • RR = Range Restriction • M = Moderator (1 = ≤ 5 yrs. Post-IPO; 2 = > 5 yrs. Post-IPO)

5. “r” - A Bivariate Correlation • “r” vs. “d” • R-square • Deriving “r” from “d,” “t,” “F-score,” “Z,” “Chi-Square” … • “r” from Incomplete Information r = Z/sqrt n if “n” = 120 and Z = 1.96 with “r” unknown then r = +/ - .179 (i.e., 1.96/10.95)

6. “r” – A Bivariate Correlation,cntd. • -17 to +17 and Enter What? • Discard the Study? • “r” and the Z-transformation? • “r” and Statistical Significance • And, a “Surprise” About Multiple Non-Significant Results

7. “r” – A Bivariate Correlationand “n” • “r” As an Independent Variable, a Dependent Variable, a Control Variable, a Moderating Variable, a Mediating Variable… • “n” – The Sample Size from which the “r” Was Calculated • To Weight the Observed Correlation in Order to Calculate the Mean Weighted Correlation Across All of the Studies • “n” and the Correlation Matrix

8. Ry (Reliability of y); Rx (Reliability of x) • Constructs vs. Observed Variables • Strategic Management Meta-Analyses with Ry = 1 and Rx = 1 • Strategic Management Variables Are Not That Good • The Choice of Ry and Rx Levels Is Counterintuitive – Lower Ry’s and Rx’s Will Improve the “Corrected r” • Ry and Rx at .8

9. RRy & RRx (Range Restriction of y and x) • Analytical Issues of Range Restriction Have Become Increasingly Complex • In Strategic Management – RRy and RRx as Deliberate Selectivity in the Sample • Strategic Management and “Survival” Issues

10. Moderation in Meta-Analysis • In Meta-Analysis a “Moderator” Is a Subgroup • Profligate Testing for Moderators • Capitalization on Chance • Loss of Statistical Power • Moderators Need Not Always Be Operationalized as a Dichotomy

11. Meta-Analytic Procedures and Results

12. Meta-Analytic Procedures and Results

13. Meta-Analytic Results:Some Diagnostics • The Magnitude of the Mean True Score Correlation • Does the 90% Confidence Interval Include Zero? Suggests that the Mean True Score Is Not Significant • Does the 80% Credibility Interval (Difference between Low and High Estimates) Exceed .11? Suggests the Existence of a Moderator • Does the % Variance Attributable to Artifacts Exceed 75%? Suggests that a Moderator Is Unlikely • And, If the Tests Had Relied on Different Rx and Ry Values? [ .7 = .48; .8 = .417; .9 = .37 ]

14. Results Summary • There is no simple relationship ( -.026, ns) between CEO equity holdings and firm financial performance. There is, however, some evidence of the existence of a moderating variable. • There is evidence of a moderating effect for time since IPO. The relationship between CEO equity holdings and firm financial performance for firms 5 years or less from IPO is .417, a significant relationship. The diagnostics suggest that a further moderating effect of this result is unlikely.

15. Results Summary, cntd. • The relationship between CEO equity holdings and firm financial performance for firms more than 5 years from the IPO is -.144, a significant relationship. The diagnostics suggest that a further moderating effect of this result is likely.

16. Other Issues in Meta-Analysis • Fixed vs. Random Effects Models • Random Effects Models – Population Parameters May Vary Across Studies • Fixed Effects Models – Population Parameters Are Invariant • “File Drawer” Problem • Unreported Null Results • “Fail Safe” Approach • The Issue Is Less a Matter of Fail Safe Algorithms than of Reliance on Too Few Studies

17. Other Issues in Meta-Analysis, cntd. • Quality of Data • Outliers • Statistical Outliers • Entry Error Outliers • Sensitivity to Outliers • The General Question of Discarding Data • Disclosure and Replicability

18. Other Issues in Meta-Analysis, cntd. • The Independence of Data • Entering Data that Are Clearly Not Independent • A Random Selection, Pooling, a Weighted “r”, a Weighted “n” • An Interesting Catch-22 • “Clearly Reflect the Same Construct” • Independence of Samples • Constructive Replication

19. General Guidelines for Meta-Analysis • There is no need to transform the input values of “r”s. • When it is necessary to impute the value of “r,” set “r” = 0. • For observed variables, rely on .8 for the reliability of the dependent and independent variables. • With observed variables, it will rarely be necessary to assign a range restriction score.

20. General Guidelines for Meta-Analysis, cntd. • Use a conservative 90% confidence interval for the meta-analysis diagnostics (for these data, 95% would be an interval of -.128 to .075, much wider than the -.112 to .059 reported). • Use a conservative 80% credibility interval for the meta-analysis diagnostics (for these data, the 90% would have been an interval of -.493 to .448, much wider than the -.389 to .336 reported). • Where the meta-analysis software provides an option, rely on a “Random Effects Model.”

21. General Guidelines for Meta-Analysis, cntd. • Assuming every effort has been made for an exhaustive search for meta-analysis input data, you need not be concerned about “file drawer” issues • Neither weight nor exclude data on the basis of the quality of the study. Instead, run two meta-analyses and compare the results for the entire data set and a reduced data set without the troublesome data

22. General Guidelines for Meta-Analysis, cntd. • Only under extremely rare conditions would there be any concerns about the independence of the data; accordingly, there is no need to combine data from separate “r”s in any manner. • No need to exclude outliers. Instead, run two meta-analyses and compare the results for the entire data set and a data set without the outliers.