The Role of Experimentation in Policy Making

1. The Role of Experimentation in Policy Making Abhijit Vinayak Banerjee IDB, September 2009

2. Theory is a very limited guide to policy • It is very good at alerting us to various possible mechanisms • It just does not tell us which of those mechanisms actually work and which don’t • Because theory is always necessarily incomplete. • It is based on concepts, which come out of our need to limit the complexity of reality • It is based on behavioral assumptions and assumptions about the environment that are untested and often untestable

3. An example of a challenging concept • Education • Let us accept the evidence that education promotes growth • How to promote education? • Concepts often used in the growth literature are things like “fraction of population completing primary school” • These are not policy levers • Policy levers are things like: • Teacher-student ratio, textbooks, school uniforms, school meals, etc. • Theory tells us nothing about which of works best • Can we assume that the policy makers know?

5. An example of unexpected behavior • The demand for health goods • Like immunization, insecticide treated bed-nets, chlorine for purifying drinking water • Standard economic theory • These goods are valuable to some people (people who live in malarial zones, say) less to others • Pricing at marginal social cost correctly discriminates between those who want them more than the social costs and those who don’t • However there are externalities: infectious diseases, intra-family issues • So some subsidy may be justified. • But no argument for giving them away • Sunk cost fallacy argues for charging a positive price.

6. The pricing of health goods • Several studies are exploring the impact of charging or subsidizing people for health behavior • First example: Lentils for vaccine. • Immunization is really low in Rajasthan (less than 5%) • What is the price elasticity of vaccine demand? Abdul Latif Jameel Poverty Action Lab

7. Testing Demand and Supply • One first possibility is that the supply channel is the problem: • Conducted monthly immunization camps in 60 villages: regular camps held rain or shine • One second possibility is that there is a problem of demand: • People not interested in immunization, scared, etc. • Can demand be affected? • Extra incentive: in 30 of the camps, provided a kg of lentils for each immunization • 60 camps’ villages remained the control group. Immunization rates were followed in treatment villages, control villages, and one neighboring village of each of the treatment villages Abdul Latif Jameel Poverty Action Lab

8. Results • Immunization coverage increased dramatically At endline: Abdul Latif Jameel Poverty Action Lab

9. High price elasticity • Surprisingly high price elasticity, in light of the expected benefits. Why might that be so: • Information? • Discount rate? • Something else? • Smokeless stoves • Chlorine for water: same effect • Bed-Nets: • Dupas and Cohen : randomize the price at which bednets are offered in pre-natal clinic • Dupas: similar experiment among households. • Deworming drugs: take up collapses with (small) positive price

10. Results: Demand Monthly Net Sales by ITN Price Abdul Latif Jameel Poverty Action Lab

11. Should People Pay for Health Goods? • Conventional economic theory: • If the health good has a high externality and is expensive, you want to subsidize its use (possibly even subsidize people to do it, like in the case of the immunization program), in particular when the price elasticity is high • “New” conventional wisdom (social marketing) • Health goods you give away may not end up in the hands of those who need it the most • The fact of paying may encourage people to use the good (sunk cost fallacy) Abdul Latif Jameel Poverty Action Lab

12. Evidence from Bed nets • Do people who get a free bed net use it less than those who have to pay? • Do people who get a free bed net need it less than those who have to pay? Abdul Latif Jameel Poverty Action Lab

13. Results: Usage Share Observed Using ITN at follow-up Abdul Latif Jameel Poverty Action Lab

14. Effective Coverage: Share of Prenatal Clients Sleeping Under ITN, by Price Abdul Latif Jameel Poverty Action Lab

15. Selection Effects of Price on Health • Do higher insecticide treated net prices induce selection of more vulnerable women (i.e. sickest)? • Result: No. People who buy nets at current cost-sharing price are healthier than the average prenatal client in the region • Ability to pay seems to be the binding factor Abdul Latif Jameel Poverty Action Lab

16. Lessons • Find no evidence that cost-sharing reduces wastage on those who will not use the product • Also find no evidence that cost-sharing induces selection of those who need net more • Cost-sharing does considerably dampen demand • As the price goes from zero to the prevailing cost-share price, uptake drops by 75% • Another example where the cost benefit analysis is not what you thought! • Free distribution seems at least as cost-effective as cost-sharing (because of externalities) • The debate generated by the paper: does this generalize? • Proponents of cost sharing proposed various theories to explain why the results may not generalize. • But they had to formulate new arguments, which can then be tested: this moved the debate beyond where it was… “cost sharing is good”. Abdul Latif Jameel Poverty Action Lab

17. Policy has to be based on learning from experience • History is a set of experiments… • Why do we need Experiments?

18. Why Experiment: First answer Evaluating impact of alternative policies is not easy • What effect (if any) did the policy have? • How would individuals who experienced the program have fared in the absence of the program? • How would those who did not experience the program have fared if they had been exposed to the program? • To know the impact of a program one must be able to answer the counterfactual: • How would an individual have fared without the program? • But one can’t observe the same individual with and without the program • Need an adequate comparison group • Individuals who, except for the fact that they were not beneficiaries of the program, are similar to those who received the program Abdul Latif Jameel Poverty Action Lab

19. Randomization • Gives us a control group that is ex ante identical to the treatment group. • Does not rely on theory to give us exclusion restrictions that then allow us to make valid comparisons • Very convenient when the theory is suspect • May not always be useful without theory • You need a priori hypotheses • May not always be useful without some econometrics. • You need some distributional assumptions

20. Why Experiment: Second answer • You typically know exactly who was treated or something close • And you control data collection • Therefore you can target your data collection. • As against in an observational study where the quasi-randomization emerges opportunistically. • You study a policy and typically decide ex post who was treated and who was not. Unless you were directly involved in the policy making—in which case why not try to randomize? • And especially with Instrumental Variables methods, you don’t always know who you are comparing • And even when you do, the actual event may be many years past • Therefore you cannot rely on data collected for you • Hence detail/creativity in data collection is lower

21. Example: Decoy visits in the Evaluation of Police Reform in Rajasthan • Can the police be taught to investigate better, treat people better and not to try to fob off complainants? • In collaboration with Rajasthan Police Department • How do you measure the willingess to register crimes • Decoy visits: Surveyors act as real complainants. • Revealed themselves if the case was about to be registered • Only 54% of complaints were registered • Neither training nor more vacations for the police nor community observers in the police station had any effect on registration • However the (random) number of previous revealed decoy visits matters: each visit increased the probability of FIR registration by 4%

22. Why experiment: Third answer • Experiments can be designed to test very specific hypotheses • Whereas observational studies, because they rely on policy variation that exists in the world, can rarely be mapped to something quite so narrow. • Therefore we can test individual elements of a policy rather than the policy as a whole

23. Technology adoption: fertilizer in Kenya (Duflo-Kremer-Robinson) • Current hike in food price underscore the needs to improve the productivity of agriculture in developing countries. • The Green revolution has changed the face of Asian Agriculture since the 1970s. Need a green revolution for Africa • Part of the solution is to develop new technologies, and part is to make sure the ones we have get used. • Big puzzle: why do so few farmers adopt fertilizer in Western Kenya • The technology is well known • It has high returns (which we confirmed in experiments) • It can be done in small quantity

24. Possible reasons • Development economists have worked on this topic for decades. Hypotheses: • Low actual returns • Lack of information/knowledge • Financing difficulties • Duflo-Kremer-Robinson set up a series of field experiments over several years in Western Kenya to investigate these issues. • Controlled trial on (randomly selected) farmers’ plot: fertilizer seem profitable

25. Information • Several experiments to understand the role and the diffusion of information: • Experiments on farmers plot were done with randomly selected farmers: we follow their adoption and that of the control farmers • Other ways to disseminate information: • Starter kits (Malawi) • Demonstration farms • Neighbors • Invited • Non invited (natural diffusion) • Bottom line: • information plays a role (10% increase in adoption among pilot farmer) • However it does not diffuse naturally (no effect of friends network) • Financial barriers remain a concern for farmers

26. Present-bias • Farmers seem to make plans to use fertilizer at harvest time, but not carry them out • By the time of planting they have no money left. • Possibly it is because they postpone the purchase until they need the fertilizer, but then they consume too much in the intervening period (may be because of family demands). • If this is the case, a program that encourages them to purchase fertilizer right after harvest, rather than later, should be effective. • With an NGO (ICS), we offer free delivery of fertilizer right after harvest (the “SAFI” program)

27. Results • SAFI is taken up by 40% of farmer and led to a 14 percentage point increase in fertilizer use (a 60% increase). • As large as the impact of a 50% subsidy • Offering free delivery at the time they need it is not sufficient (it led to a 7 percentage point increase in use). • When farmer are asked to chose the timing, they chose the early delivery.

28. What can’t experiments do? • Tell you what exchange rate to set • But it is not clear that regressions of any kind are the right methodology for answering such questions • May better to be build simulation models based on reliable parameter estimates. • From micro regressions. • Experiments may play an important role here

29. The question of external validity • How do we know that the result from an experiment will generalize to other places • Problem for all forms of empirical research. • However it might seem that it is less of a problem for cross-country comparisons • Because the result is an average covering a wide range of environments • Not exactly right • Different results when you include different controls • Could be because the set of countries being compared are different • One advantage of experiments is you are much more likely to know the exact population that is being compared.

30. However • There is no dismissing this issue. • Replications help build some confidence in the results • Theory helps give us identify the kind of replications that make sense. • The problem remains that as of now organizations that participate in randomized trials tend to be “strong” organizations • We need more randomized evaluations in weak organizations

31. Most importantly • There is an infinity of policy questions out there. • Textbooks, uniforms, meals, teachers, blackboards, toilets, scholarships, computers.. • Theory is the only reliable guide to what are the right questions to answer • Unfortunately in many cases we do not have a useable theory: that tells us, for example, textbooks are more like computers or more like teachers.

32. On balance however • We have learnt things from experiments we could have not learnt otherwise • The benefits from the experimental approach go beyond the specific learnings (with their limitations). • Have helped popularize a culture of emphasizing experimentation and learning from experiments • And have imposed some discipline on what can and cannot be claimed as known • Does not mean that we learn only from Randomized Experiments • But experiments set a standard of evidence and a culture of being willing to embrace failures.

33. THANK YOU!