‘What works in development’? A critical view on ‘randomization in the tropics’

1. ‘What works in development’? A critical view on ‘randomization in the tropics’ Carlos Oya SOAS, London SOAS 14 December 2011

2. What is development? Back to basics • Improvement in material living standards (income per capita) • Being educated and living a longer and healthier life • ‘Freedom’ • transformation of the productive structure (and the capabilities that support it) and the resulting transformation of social structure: dissolution of the traditional family, changes in gender relationships, rise of labour movement, the advent of the welfare state, etc.... All through, for example, industrialization

3. How can aid support development? • By contributing to the improvement of education and health outcomes • By contributing to economic infrastructure • By contributing with ideas for development • By reforming institutions for development • By providing individuals/households with tools for better livelihoods (microcredit...) • By ‘solving problems’ • By helping states finance economic transformations • And so on

4. How is aid evaluated? • Logical frameworks (process and outcome), often loathed but still widely used • Macro impact through cross country regressions (outcome monitoring): • Growth = A + a*Aid + b*X (+c(Aid*policy)) • Poverty incidence = A + a*Aid + b*X • Micro impact through experiments: • (Social) Experiments (RCTs) • Quase-experiments (incl. Instrumental variables)

5. Leading questions: the macro-micro tension • Can foreign assistance raise growth rates and eliminate poverty?  cross-country regression analysis • What sorts of projects are likely to be effective? OR what works in development?  experimental design (RCTs) • Long-run macro effects vs what difference does a particular intervention make

6. Basic flaws for starters • Reductionism and ‘problem solving’ – ‘randomization is only feasible for a non-random subset of the interventions and settings relevant to development’ (Ravallion 2009) • Simplification of causal mechanisms and relations  linearization of processes • Not enough (or no) theory • Concealed ideological/methodological assumptions: e.g. methodological individualism, anti-state bias, etc.

7. The tragedy of ‘problem solving’ ‘I am in charge of redecorating our bathroom while my partner is away. The paint is peeling and there is mildew on the ceiling above the shower. To demonstrate I got value for our money I will get two quotations for the redecoration. Many donor governments are treating the complex problems of poverty like my bathroom. They contract a Third Party Operator to deliver a result pre-determined by DFID. At the end of three or four years, there is an evaluation to check on results before paying the contractor’ R. Eyben in Green’s blog

8. “Britain has given the world Shakespeare, Newtonian physics, the theory of evolution, parliamentary democracy—and the randomized trial.” Deaton (2010: 438, quoting British Medical Journal)

9. RCTs in the tropics What is an RTC? What are the main advantages? Dealing with selection bias and endogeneity Moving from correlation to causality Focusing on intervention and addressing attribution problem Solves problems in other comparable methods (IV, propensity score matching, etc.) Apparent simplicity in interpretation of results • A comparison of like-with-like that only differ because one group receives intervention and one doesn’t – a social experiment where some units are randomly assigned an intervention while the rest form the control group, and we compare average outcomes of the two groups • A kind of with vs without comparison • Can be cross-sectional of panel-longitudinal

10. What’s the problem? Technical level • Conditions to find appropriate control group are difficult to achieve in development contexts – quase-randomization very common • Medical trial ‘gold standard’ not applied: no triple blinding (participant, agent, analyst) • Evaluation has to be designed ex ante evaluation affects intervention design • Logistical complications when several treatments must be evaluated in combination • Dealing with heterogeneity of agents/units within groups - only estimated average treatment effect for all  distribution of benefits? Data mining in ex-post subgroup analysis • Dealing with spill-over effects on neighbours • Challenges in scaling-up – comparability of RCTs

11. What’s the problem? Conceptual/epistemological level • Lack of theory  but exploratory analysis may be useful • Tells us what works, but not why or how  response: RCTs only tackle part of the evaluation effort (White) • Reductionism in scope and applications (though these are increasing fast)  External validity  transferability problem; RCT results cannot be extrapolated outside the specific trial context • Relevance? Most devt. programmes target certain groups so randomized assignment is inadequate • There is no ‘gold standard’ method (Cartwright) – simple induction is not better in social science than in natural science • Asymmetry between (a) the rigour to attain internal validity and (b) the loose approach to defend transferability • ‘Development’ trivialized and simplified: Hamlet without Prince of Denmark (see below)

12. A basic but central epistemological problem(N. Cartwright) Three distinct causal claims 1. It works somewhere. 2. It works in general. 3. It will work for us. RCTs can only score well (if ideally implemented) in (1)… and it cannot even tell you why it worked and for whom within groups

13. Ethical considerations • Having an untreated group  How/what to inform/deal with control group if double blinding is not used • Managing expectations of people (treated and untreated) • Possible contamination of treatments by new associated interventions  Political considerations • Scope for data mining or selective reporting of findings if designed protocols not peer reviewed and published ex ante • However: ‘The really unethical thing is the spending billions of dollars each year on programmes that don’t work’ (White 2011)

14. Hamlet without the Prince of Denmark (HJ Chang) • ‘‘development’ has come to mean poverty reduction, provision of basic needs, individual betterment, sustenance of existing productive structure – that is, anything but ‘development’ in the traditional sense.’ (Chang 2010) • ‘doing more of the same thing in terms of one’s productive activities is not how today’s developed countries have become developed.’ • ‘mainstream development discourse sees these increases in productive capabilities as happening mainly through individual betterment.’  Development then is reduced to what happens at micro-level with interventions that can be effectively randomized

15. Concluding remarks on key risks with RCTs • A new fad, which will impact on research and interventions with unintended consequences via funding priorities • Researchers may lose interest in important questions (historical, institutional, and structural ) that cannot feasibly be explored using randomization methods • Tendency towards focus on short-run impact of small projects away from long-run combined sector-macro policies • Method (and its boundaries) drives concept and questions • Losing interest in theory and history