Meta analysis The case-study of aid effectiveness European Journal of Political Economy. 1 st issue 2008 p 1-24

1. Meta analysisThe case-study of aid effectivenessEuropean Journal of Political Economy. 1st issue 2008 p 1-24 Martin Paldam http://www.martin.paldam.dk Click on to Working Papers Joint work with Hristos Doucouliagos, Deakin university, Melbourne, Australia

2. Primary studies of to estimate an effectData: The primary dataYour work in ½-1 year: Less than 1 man-year of work. Reflects: You ideas + have read.Meta studies of the literature on an effectData: The literatureThe AEL-AAL 250 studies: 200 man-years of work.Reflects: The ideas of everybody in the field.Forensic economics: Quantitative studies of litera-ture: Where are we? and how did we get there?

3. The talk discusses the political economy of some of the results of 8 meta studies of: AEL, Aid Effectiveness Literature, 2005-7 4 studies: 2 out ,1 accepted and 1 R&RAAL, Aid Allocation Literature 2007-9. 2 new WPs, 2 in partly first draftMeta studies often give embarrassing results. Strong reactions of referees: Most negative and positive I have experienced.

4. M1 studies of the effect μ, giving M2 > M1 estimates of μ, that can be calibrated to same scale (partial correlations). The full-set, [M]F, and the best-set, [M]B.Coded with a C-vector for each estimate of all possible characteristics: Our case: An C-vector for 60 possible model controls, estimation methods, data, author, journalFunnel plots, MRAs, …

5. Meta studies: Three questions to a literature 1. Does the result converge to something that we can consider the true value?2. Are there breakthroughs (structural jumps) which can be identified and explained: Does journal quality, estimators, etc matter?3. Does the distribution of the results point to biases

6. Three general results (all embarrasing):(a) Amazing variation in results: Always signifiantly different results(b) Estimators rarely important: Profession obsessed with estimators? Sociology of our profession: Macho test (c) Publication biases common: Publication biases: Look at funnel plot. All estimates plotted over precision ln N

7. Case: The price elasticity of beer. Intuition: Negative but not large. Literature: Average finding surprisingly large

8. Study the funnelsFATs testing for asymmetriesCorrect distribution to find true average Several formulas, fortunately robust!Tom Stanley (+ Chris Doucouliagos)New paper in Oxford Bulletin Monte Carlo experiments

9. The aid effectiveness discussion Great subject for Meta study: • Emotional: One side is the side of the angels! • Strong interests: Stay on the gravy train! • A basic statistical fact that has to be overcome The zero-correlation result Zero-correlation result  two great games: The control variables game: Find a plus set The causality game: Find a big negative bias!

10. Zero correlation between aid and growth: for all aid recipients

11. The causality game: A literature on each arrow

12. The reverse causality flow R.We wish a negative coefficient

13. Hence: The data is a big problem. They point to aid ineffectiveness and it is not obvious that it is due to simultaneityMy own primary studies: very little(everybody know)Shift perspective: to research

14. Point of talk: Economics assumes that all humans have priors/interests. Why not us?Also economists also have priors/interests. We operate on the market for economic research. It may not be a perfect market.Our small talk at lunch tables, in bars etc. Often assumes that journals have biases, that referee processes are …

15. Limitation of discussion: (a) Empirical + (b) macroAd (a): We analyze M studies of the same effectAd (b): Data is limited relative to the amount of research. Thus data mining problemWhat can we prove? Meta studies claim they can prove a great deal.Not for individual studies, but for specific literatureOur studies typically find asymmetries pointing to priors. (In a moment)

16. We like to believe that research is a process that search for truth that converges to the truth.Assume: Truth is the true value of μ.Process in individual/paperProcess on market, market incentives: Are the incentives truth-finding consistent?

17. Process for individual researcher, XX search for a value of μ, till he is satisfied. The paper is thus the result of a stopping rule in Xs search process:X stops when he has found a μ that:(a) Is in accord with his priors or his interests(b) Is publishable (c) Can be defended statisticallyThus: When I stop have I found truth or confirmed my priors?

18. Process on market:Innovation + replication generates trust in resultsInnovation: Theory, estimation technique, data.Innovation easy to publish (?)Replication: Independent: Other authors on new data Dependent (1): Same author on new dataDependent (2): New author on new data Macro: Normally overlapping data so only: Marginally independent Thus: Effect on estimate of extra data

19. Data mining:The number of estimates on subsets of the same data is large relative to the number of observationsEx: Phillips curves. Estimated on the w, p, u data for 30 countries over the last 50 years? Guess: 5 mio estimated? Ex: Money demand,…Ex: Growth regressions: Sala-i-Martin alone about 90 million

20. Consequence:Type I errors reduced: Rejecting true modelType II errors increased: Accepting false modelsHence in heavily mined fields: There must be many type II errorsThus independent replication necessary. And meta studies highly needed

21. Not problem of each researcher; but the collective. We all read up some of the literature and join the mining collective. We fish in the common pond of df ’s. A double tragedy of the common.1. It is the standard tragedy that we exhaust the df.2. It is also a tragedy that nothing visible happens we can just go on and on!

22. The Aid Effectiveness Literature,AEL, studies:μ = ∂g/∂h, conditional on everything our pro- fession has thought of – it is a great del.Micro base. Average LDC growth 1.5%. Projects based on cost-benefit (growth contribution): Social rate of return 10%. Thus, h = H/Y ≈ 7.5%  0.75 pp growthIt should be highly visible in data, but as we have seen it is not.

23. Thus a puzzle: It is the AEL paper generator:In 2006 aid exceeded $ 100 bill + AEL paper nr 100 came out. No agreement on results.Data: Aid started in mid 1960s. Now ap 145 data per year: and 6000 annual observations. Average to 5 years: about 1000.Published regressions 1,025, made 25,000? Alternative: Sum of N is 25,000 Likely that false models have appearedBefore we look at results look at the data,

27. Thus, the AEL starts from a zero correlation, and put structure on this result till something appears.Model is the same as the Barro-growth regression:git = α + μhit + (γx1it + ... + γxnit) + uit orgit = α + μhit + δzit + ωhitzit + (γx1it + ... + γxnit) + uitResearchers have tried 60 x’es and 10 z’sMany millions possible permutations, each gives a different estimate of μ. As average is zero half are positive, and 5% are significant. What to choose?

28. A tool: The funnel plot

30. In the AEL: Everything goes together to generate:The reluctancy bias Researchers and journals are reluctant to publish negative resultsProof followsLet us look at the 5 priors – one at a time:

31. Polishing:We want to display our goods as well as possible.Then they are easier to sell to journalsCareer + feel well. Strong incentives:

33. What do we expect to see?Easier to polish in small samples: Study t-ratios as a function of df: t = t(N)If random ln t proportional to ln N: The MST: ln│ti│= α0 + α1 ln Ni + ui test: α1 < 0  polishingOften found in meta studies, in the AEL also.

34. Ideology:An ideology that predicts the size (sign) of μ authors with that ideology find that size (sign).In the AEL:a. Libertarians (Friedman, Bauer): Aid  larger public sectors  planning  socialism  harmsb. “New-left”: Aid from capitalist states  capita-lism and exploitation  harmsBoth OK (not many authors) especially early

35. Goodness:Common finding: We all want to look good and most want to be politically correctShown as asymmetry of funnel plot: The FAT. Part of the funnel is missing.In the AEL: Aid aims at doing good (+ …) So to show that it fails is bad. Nice to be on the side of the angels: Bono, Jeff Sachs, Gordon Brown, Koffi Anan, etc.

36. Causes reluctancy. The FAT: εi = β0 + β1si + υi, where εi is the standardized effect, and siis its standard error. Also, look at the funnels We look at: μ = μ(N) and μ = μ(t)in a moment

37. Interests:Normally Ok: there are many interests. In the AEL: diffuse interests on the one side.And the “Aid Industry” on the other side.It has a turnover of $ 100 bill. It has a composition with includes a bureaucracy + political parties + NGOs + business + unions.It gives about 10% in consultancy fees + 0.25-0.5 % to research.

38. The aid industry want aid to workMany of those working in the AEL are working for/financed by the aid industryProblem: Many does not write so!Gives reluctancy as wellPsychology: Alignment of priors to interestsObs: Goodness + interests give same result in this case. Hence we expect clear effects.

39. History:50% are in one paper only. The rest are in more + many additional links. People are z > 0.5 committed after one paper to find the same result. Our guess z = 0.9Also, same department as, writing PhD under, …Very significant: Fighting schools

40. Reluctancy:Asymmetry of missing negative valuesHow should it look? Should be visible on μ = μ(N). Sorting out μ = μ(N) and μ = μ(t)Problem: Learning by doing: μ = μ(t) should slope upward.Look at the two graphs:

43. Problem: No learning by doing, See graphical interpretation, next slideRun: μNt = α + β ln N + γ t + εMulticollinearity: N goes up for t rising

45. Thus:Reluctancy confirmed: Is it goodness or interests? Test: Use (poorly measured) interest variableEffect of interest: Always sign as expected, not always significant:It is not a big effect, but it is there!

46. Correct funnel for asymmetry: Net result insignificantThus the literature has not showed that aid works after 100 papers over 40 years.DepressingSet of MRAs – very bulky see paper! I just give some highlights:

48. The end:We behave as predictedby our theoriesAlso economists are human!