1 / 48

Martin Paldam martin.paldam.dk Click on to Working Papers

Meta analysis The case-study of aid effectiveness European Journal of Political Economy. 1 st issue 2008 p 1-24. Martin Paldam http://www.martin.paldam.dk Click on to Working Papers Joint work with Hristos Doucouliagos , Deakin university, Melbourne, Australia.

mimir
Download Presentation

Martin Paldam martin.paldam.dk Click on to Working Papers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Meta analysisThe case-study of aid effectivenessEuropean Journal of Political Economy. 1st issue 2008 p 1-24 Martin Paldam http://www.martin.paldam.dk Click on to Working Papers Joint work with Hristos Doucouliagos, Deakin university, Melbourne, Australia

  2. Primary studies of to estimate an effectData: The primary dataYour work in ½-1 year: Less than 1 man-year of work. Reflects: You ideas + have read.Meta studies of the literature on an effectData: The literatureThe AEL-AAL 250 studies: 200 man-years of work.Reflects: The ideas of everybody in the field.Forensic economics: Quantitative studies of litera-ture: Where are we? and how did we get there?

  3. The talk discusses the political economy of some of the results of 8 meta studies of: AEL, Aid Effectiveness Literature, 2005-7 4 studies: 2 out ,1 accepted and 1 R&RAAL, Aid Allocation Literature 2007-9. 2 new WPs, 2 in partly first draftMeta studies often give embarrassing results. Strong reactions of referees: Most negative and positive I have experienced.

  4. M1 studies of the effect μ, giving M2 > M1 estimates of μ, that can be calibrated to same scale (partial correlations). The full-set, [M]F, and the best-set, [M]B.Coded with a C-vector for each estimate of all possible characteristics: Our case: An C-vector for 60 possible model controls, estimation methods, data, author, journalFunnel plots, MRAs, …

  5. Meta studies: Three questions to a literature 1. Does the result converge to something that we can consider the true value?2. Are there breakthroughs (structural jumps) which can be identified and explained: Does journal quality, estimators, etc matter?3. Does the distribution of the results point to biases

  6. Three general results (all embarrasing):(a) Amazing variation in results: Always signifiantly different results(b) Estimators rarely important: Profession obsessed with estimators? Sociology of our profession: Macho test (c) Publication biases common: Publication biases: Look at funnel plot. All estimates plotted over precision ln N

  7. Case: The price elasticity of beer. Intuition: Negative but not large. Literature: Average finding surprisingly large

  8. Study the funnelsFATs testing for asymmetriesCorrect distribution to find true average Several formulas, fortunately robust!Tom Stanley (+ Chris Doucouliagos)New paper in Oxford Bulletin Monte Carlo experiments

  9. The aid effectiveness discussion Great subject for Meta study: • Emotional: One side is the side of the angels! • Strong interests: Stay on the gravy train! • A basic statistical fact that has to be overcome The zero-correlation result Zero-correlation result  two great games: The control variables game: Find a plus set The causality game: Find a big negative bias!

  10. Zero correlation between aid and growth: for all aid recipients

  11. The causality game: A literature on each arrow

  12. The reverse causality flow R.We wish a negative coefficient

  13. Hence: The data is a big problem. They point to aid ineffectiveness and it is not obvious that it is due to simultaneityMy own primary studies: very little(everybody know)Shift perspective: to research

  14. Point of talk: Economics assumes that all humans have priors/interests. Why not us?Also economists also have priors/interests. We operate on the market for economic research. It may not be a perfect market.Our small talk at lunch tables, in bars etc. Often assumes that journals have biases, that referee processes are …

  15. Limitation of discussion: (a) Empirical + (b) macroAd (a): We analyze M studies of the same effectAd (b): Data is limited relative to the amount of research. Thus data mining problemWhat can we prove? Meta studies claim they can prove a great deal.Not for individual studies, but for specific literatureOur studies typically find asymmetries pointing to priors. (In a moment)

  16. We like to believe that research is a process that search for truth that converges to the truth.Assume: Truth is the true value of μ.Process in individual/paperProcess on market, market incentives: Are the incentives truth-finding consistent?

  17. Process for individual researcher, XX search for a value of μ, till he is satisfied. The paper is thus the result of a stopping rule in Xs search process:X stops when he has found a μ that:(a) Is in accord with his priors or his interests(b) Is publishable (c) Can be defended statisticallyThus: When I stop have I found truth or confirmed my priors?

  18. Process on market:Innovation + replication generates trust in resultsInnovation: Theory, estimation technique, data.Innovation easy to publish (?)Replication: Independent: Other authors on new data Dependent (1): Same author on new dataDependent (2): New author on new data Macro: Normally overlapping data so only: Marginally independent Thus: Effect on estimate of extra data

  19. Data mining:The number of estimates on subsets of the same data is large relative to the number of observationsEx: Phillips curves. Estimated on the w, p, u data for 30 countries over the last 50 years? Guess: 5 mio estimated? Ex: Money demand,…Ex: Growth regressions: Sala-i-Martin alone about 90 million

  20. Consequence:Type I errors reduced: Rejecting true modelType II errors increased: Accepting false modelsHence in heavily mined fields: There must be many type II errorsThus independent replication necessary. And meta studies highly needed

  21. Not problem of each researcher; but the collective. We all read up some of the literature and join the mining collective. We fish in the common pond of df ’s. A double tragedy of the common.1. It is the standard tragedy that we exhaust the df.2. It is also a tragedy that nothing visible happens we can just go on and on!

  22. The Aid Effectiveness Literature,AEL, studies:μ = ∂g/∂h, conditional on everything our pro- fession has thought of – it is a great del.Micro base. Average LDC growth 1.5%. Projects based on cost-benefit (growth contribution): Social rate of return 10%. Thus, h = H/Y ≈ 7.5%  0.75 pp growthIt should be highly visible in data, but as we have seen it is not.

  23. Thus a puzzle: It is the AEL paper generator:In 2006 aid exceeded $ 100 bill + AEL paper nr 100 came out. No agreement on results.Data: Aid started in mid 1960s. Now ap 145 data per year: and 6000 annual observations. Average to 5 years: about 1000.Published regressions 1,025, made 25,000? Alternative: Sum of N is 25,000 Likely that false models have appearedBefore we look at results look at the data,

  24. Thus, the AEL starts from a zero correlation, and put structure on this result till something appears.Model is the same as the Barro-growth regression:git = α + μhit + (γx1it + ... + γxnit) + uit orgit = α + μhit + δzit + ωhitzit + (γx1it + ... + γxnit) + uitResearchers have tried 60 x’es and 10 z’sMany millions possible permutations, each gives a different estimate of μ. As average is zero half are positive, and 5% are significant. What to choose?

  25. A tool: The funnel plot

  26. In the AEL: Everything goes together to generate:The reluctancy bias Researchers and journals are reluctant to publish negative resultsProof followsLet us look at the 5 priors – one at a time:

  27. Polishing:We want to display our goods as well as possible.Then they are easier to sell to journalsCareer + feel well. Strong incentives:

  28. What do we expect to see?Easier to polish in small samples: Study t-ratios as a function of df: t = t(N)If random ln t proportional to ln N: The MST: ln│ti│= α0 + α1 ln Ni + ui test: α1 < 0  polishingOften found in meta studies, in the AEL also.

  29. Ideology:An ideology that predicts the size (sign) of μ authors with that ideology find that size (sign).In the AEL:a. Libertarians (Friedman, Bauer): Aid  larger public sectors  planning  socialism  harmsb. “New-left”: Aid from capitalist states  capita-lism and exploitation  harmsBoth OK (not many authors) especially early

  30. Goodness:Common finding: We all want to look good and most want to be politically correctShown as asymmetry of funnel plot: The FAT. Part of the funnel is missing.In the AEL: Aid aims at doing good (+ …) So to show that it fails is bad. Nice to be on the side of the angels: Bono, Jeff Sachs, Gordon Brown, Koffi Anan, etc.

  31. Causes reluctancy. The FAT: εi = β0 + β1si + υi, where εi is the standardized effect, and siis its standard error. Also, look at the funnels We look at: μ = μ(N) and μ = μ(t)in a moment

  32. Interests:Normally Ok: there are many interests. In the AEL: diffuse interests on the one side.And the “Aid Industry” on the other side.It has a turnover of $ 100 bill. It has a composition with includes a bureaucracy + political parties + NGOs + business + unions.It gives about 10% in consultancy fees + 0.25-0.5 % to research.

  33. The aid industry want aid to workMany of those working in the AEL are working for/financed by the aid industryProblem: Many does not write so!Gives reluctancy as wellPsychology: Alignment of priors to interestsObs: Goodness + interests give same result in this case. Hence we expect clear effects.

  34. History:50% are in one paper only. The rest are in more + many additional links. People are z > 0.5 committed after one paper to find the same result. Our guess z = 0.9Also, same department as, writing PhD under, …Very significant: Fighting schools

  35. Reluctancy:Asymmetry of missing negative valuesHow should it look? Should be visible on μ = μ(N). Sorting out μ = μ(N) and μ = μ(t)Problem: Learning by doing: μ = μ(t) should slope upward.Look at the two graphs:

  36. Problem: No learning by doing, See graphical interpretation, next slideRun: μNt = α + β ln N + γ t + εMulticollinearity: N goes up for t rising

  37. Thus:Reluctancy confirmed: Is it goodness or interests? Test: Use (poorly measured) interest variableEffect of interest: Always sign as expected, not always significant:It is not a big effect, but it is there!

  38. Correct funnel for asymmetry: Net result insignificantThus the literature has not showed that aid works after 100 papers over 40 years.DepressingSet of MRAs – very bulky see paper! I just give some highlights:

  39. The end:We behave as predictedby our theoriesAlso economists are human!

More Related