1 / 28

A Quarter Century on, Where are we?

A Quarter Century on, Where are we?. Tom Stanley, Hendrix College.

juliette-bernard
Download Presentation

A Quarter Century on, Where are we?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Quarter Century on, Where are we? Tom Stanley, Hendrix College (MRA) is at once a framework in which to organize and interpret exact and inexact replications, to review more objectively the literature and explain its disparities, and to engage in the self-analysis of investigating the socioeconomic phenomenon of social scientific research itself– Stanley & Jarrell (1989, p. 168). T.D. Stanley, Hendrix College MAER-Net September 12, 2014 Stanley, T.D. and S.B. Jarrell (1989) Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 3: 161-70

  2. Where are we going? Exponential @ 18%/year

  3. Has MRA’s promise been realized? • More objective reviews of economic research • Explanation of disparate research findings • Investigation of the socio-economics of economics research? T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  4. Investigation of the socio-economics of economics research? • Female economists find less wage discrimination against women than do male researchers (S&J, 1998; J&S, 2004, Weichselbaumer & Winter-Ebmer, 2005) • Publication bias is the result of professional incentives and the pressure to publish. • Researcher ideology affects reported results (Doucouliagos and Paldam, 2006). • The Research Cycle: Reported findings generally confirm a novel hypothesis; later, the incentive for rejection increases (S, J & D, 2008). T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  5. Has MRA’s promise been realized? • More objective reviews of economic research • Explanation of disparate research findings • Investigation of the socio-economics of economics research, . . . but much more to do. • Framework to organize and interpret exact and inexact replications? T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  6. Framework to organize and interpret exact and inexact replications? • Deakin University’s (Chris Doucouliagos) meta-data repository. • Bob Reed, Maren Duvendack, and others have expressed interest in organizing something more formal. • Piers Steel and metaBUS. • Needless to say, . . . . T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  7. How about S&J’s (1989) exact MRA model? Does anyone still use it? bi = b + SbkZik +ei(3) Where: “biis the reported estimate of b from the ith study. . . Zik the meta-independent variable which measures relevant characteristics of an empirical study andexplains its systematic variation” (p. 164) • Zikmight include: • Dummy variables for omitted variables. • Specification variables • Sample Size • Author characteristics • Data characteristics T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  8. Due to obvious Heteroskedasticity, S&J 89 recommended the WLS version of (3): ti= bi/Sbi = b (1/Sbi) + Sbk (Zik/Sbi) +ei/Sbi(4) • t-value, ti, is the dependent variable and precision(1/Sbi) is an independent variable. • WLS is neither fixed- nor random-effects,in practical application, WLS is better than both (Stanley and Doucouliagos, 2013a&b, Deakin SWP). • We never wanted to use fixed- or random-effects, in spite of citing Hedges and Olkin (1985). • Had we included the intercept, we would have fully anticipated current practice {FAT-PET-MRA}. T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  9. Neither Fixed • Fixed-effect MRA: same as our WLS regression, but divides SEs by square root of MSE (H&O, 85) • causes SEs & CIs to be too small & too narrow. • assumes that policy makers wish to make inferences to a population that is identical,in every respect, to past research. Like that happens! • So why divide by root MSE??? • WLS already correct the SEs for both excess heterogeneity and heteroskedasticity • Fixed-effect MRA is never relevant in economics. T.D. Stanley, He n drix College MAER-Net September 12, 2014

  10. Nor Random • Random-effects MRAadds a second error term, ni , to the conventional meta-regression model, bi = b + SbkZik+ni+ ei(1) Where niis assumed to be normal and independent of the sampling errors, ei, and the moderators, Zik. • Problems: • In economics, excess heterogeneity is systematic! • Typically, ni will be the result of omitting relevant variables; thus, it introduces bias. • In the 1980s, we saw no reason to use FE or RE • Now, we know that we should never use FE or RE! T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  11. Problems and Issues—2014 Multiple Meta-Regression (MRA) • Should we divide meta-regression SEs by the square root of MSE? No! (Hedges and Olkin, 1985) • Does Random-Effects MRA become biased with publication bias? YES!(Stanley and Doucouliagos, 2013b) Meta-analysis (MA)—Weighted Averages • Fixed-Effects Estimator (FEE): confidence intervals are too narrow if there is heterogeneity. • Random-Effects Estimator (REE): can be very biased with publication bias {already widely established} T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  12. Neither Fixed nor Random ̶ 2014 • In two recent simulation papers, Chris and I show: • WLS is as good as Fixed or Random-Effects under the best conditions for FE and RE. • If we are making inferences to policy settings or to future research, WLS is much better than FE. • When there is publication bias, WLS is much better than RE. • Worse still: all tests of either heterogeneity or publication bias are known have low power. • Therefore, in practice, we should never use either FE or RE. . . . Never! T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  13. Our two working papers: Stanley, T.D., Doucouliagos H(C). 2013a. Neither Fixed nor Random: Weighted least squares meta-regression analysis. SWP, Economics Series 2013-1, Deakin University. http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2013_1.pdf Stanley, T.D., Doucouliagos H(C). 2013b. Better than Random: Weighted least squares meta-regression analysis. SWP, Economics Series 2013-2, Deakin University. http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2013_2.pdf T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  14. Weighted Least Squares (WLS-MRA) • MRAs should never be estimated by OLS, because there is much variation among the reported SEs of bi or effecti • Enter WLS: =(MtW-1M)-1MtW-1b (2) Where: = T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  15. The Gauss-Markov Theorem{Proportional Heterogeneity Invariance} • As long asWin (2) is known up to some unknown proportion, s2, WLS{ } is the Best {Minimum Variance} Linear Unbiased Est. • Invariance to proportional excess heterogeneity is a robustness property of the Gauss-Markov Theorem and WLS. • It is not an assumption. T.D. Stanley, Hendrix College SRSM July 2, 2014

  16. Traditional, Unrestricted WLS replaces with: = (5) and is estimated by the WLS residuals, automatically T.D. Stanley, Hendrix College SRSM July 2, 2014

  17. Simulation Design • Generate Yjand estimate bfrom: Yj= 100 + b X1j+a2X2j+a3i X3j+ ej(6) • Half the studies omit the relevant variable X2i • a3i~N(0, ) adds excess random heterogeneity by always omitting relevant variable X3i • When b= 1, r between Y and X1 is .27 • n= {62, 125, 250, 500, 1000} in primary regressions • X1j , X2j, X3j are generated randomly, but X2j & X3j are forced to be correlated with X1j . • Fixed- or random-effects model is always true. T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  18. Simulation Design—Cont. • Experiment 1: 10,000 WLS, RE and FE-MRAs are calculated with one moderator variable, Mi= {0,1}, reflecting whether the original study omitted X2i, or not, (Tables 1 & 2) bi = b0+ b1Mi+ ui(7) Where: bi is the ith primary study’s estimate of b • Experiment 2: Experiment 1 plus 50% of the studies select statistically significant results, and either MRA (7), above, or a multiple FAT-PET-MRAis used. (Tables 3 & 4) bi = b0+ b1Mi+ b2SEi + ui (8) T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  19. Table 1: Coverage Percentages T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  20. Table 2: Bias and MSE T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  21. Simulation Results I:“A house divided by itself cannot stand”—A. Lincoln • FE-MRA: unacceptableSEs in most actual applications. Thus, Do Not Divide by root MSE! • When there is no heterogeneity and FE-MRA is true, WLS-MRA is equivalent to FE-MRA. • RE-MRA: When RE-MRA’s model is true, WLS-MRAprovides acceptable and comparable coverage, and its bias and MSE are a bit better. • Irony:WLS-MRA dominates RE-MRA in those exact circumstances for which RE-MRA is designed— large additive, excess random heterogeneity T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  22. Table 3: Bias and MSE with 50% Publication Bias T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  23. Table 4: Bias, MSE (50% Pub’bias) FAT-PET-MRA T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  24. Simulation Results IIWhen there is publication bias: • WLS-MRA dominates RE-MRA. • WLS-MRA always has less bias and, on average, substantially lower MSE • Irony:WLS dominates RE in those exact circumstances for which RE-MRA is designed. • When RE is better, it is not much better, and those cases cannot be identified, in practice. • Thus, there is No Reason to use Random-Effects Meta-Regression. . . . . . Ever! T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  25. Why WLS works so well when there is high heterogeneity? For very large heterogeneity, the random heterogeneity term, , will dominate sampling error, , and its variation, making the overall variance of the estimate, , roughly proportional to .

  26. Unlike S & J, S & J’s MRA ModelStill works after all these Years Multiple MRA • Should we divide MRA SEs by √MSE? Never! • Is RE-MRA biased with publication bias? Yes! • WLS-MRA dominates RE-MRA with or without Correcting for Publication Bias. • WLS also dominates REE and FEE weighted averages when combining Cohen’s d from RCTs. T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  27. The Task Ahead • We need more realistic simulation studies • Alternative modeling strategies {general-to-specific; Bayesian modeling} • Unbalanced panel MRA models. • We need to continue to raise the quality of MRA applications, making them more robust, comprehensive and rigorous. • Wish us luck at ASSA in Boston. T.D. Stanley, Hendrix College MAER-Net September 12, 2014

  28. Thank You!

More Related