Benefits and limits of randomization

1. Benefits and limits of randomization 2.4

2. Tailoring the evaluation to the question • Advantage: answer the specific question well • We design our evaluation to answer exactly the question we want to answer, • we choose the place and time • Disadvantage: one evaluation only answers the specific question(s) it was designed to answer • One treatment and comparison group, creates one difference so usually answers only one question

3. General vs. specific research • Indonesia family life survey • Survey on education, health, assets. Representative of 83% of Indonesia, 4 waves since every 1993 • Use to assess shocks or programs where happens to be some quasi randomness in implementation • Used for evaluating school building program, (Duflo, 2000) • Community monitoring evaluation in Indonesia • Communities randomized to receive either more external or more local monitoring of road projects • Know from start we can compare these two approaches to monitoring • Collect exactly the data needed to monitor corruption by digging up random sample of road and weigh materials used

4. Few assumptions, transparent findings • We know there is no selection bias, • only systematic difference between treatment and comparison groups is the program • Fewer assumptions than for quasi randomized • that we have controlled for all differences • that the formula for administering the program cut off was strictly applied • Direct comparison between two groups gives us our basic result • less complex analysis helps transparency of findings

5. Prospective evaluation • What is a prospective evaluation? • Evaluation designed in advance • Advantages • Collect specific data • Collaborative design and evaluation • Disadvantages? • Long term results emerge in the long run • Q: What approaches could give us long run results in the short run?

6. When is a randomized evaluation not useful? • When outcomes can only be measured at a very high level • Fixed vs. floating exchange rates • Freedom of the press • When general equilibrium effects are important • Outcome (like price) is determined by the aggregate of hundreds or thousands of interactions • If we change some interactions but not others, we can only observe the aggregate effect • Only possible to study when markets are somewhat separate

7. Example: job counseling in France • Unemployed youth were given job counseling • Those given counseling found jobs sooner than those not given counseling • But what was the overall effect on youth employment: did the counseled youth just displace other youth? • Randomized how many youth counseled by city • Where more youth counseled, uncounseled did worse, i.e. displacement was a real problem Crepon et al 2013

8. Ethical considerations: background • Belmont principles (and equivalent in other countries) establish ethical rules for research. Include 3 key principles: • Respect for persons: participants should be informed of risks and given a choice about participation • Benefice: the risks of research should be carefully weighed against the benefits. Risks should be minimized • Justice: the people (and the type of people) who take the risks should be those who benefit

9. Implications: approval for study • Many universities have Institutional Review Boards who review all human subject research of students, faculty and staff • May also require approvals from review board where study takes place • Approval needed before study starts • Good to get general approval at conceptual stage to cover piloting • Provide detailed surveys and final numbers of subjects covered before study launch • Approvals usually need to be updated every year with updates of people reached and any adverse consequences

10. Respect for persons • Informed consent must be gained from participants in the study • Waivers given if minimal risk and cost of collecting consent is high, e.g. knowing they are part of study will change behavior substantially • Particular care needed for those who might not be able to judge risks well or find it hard to say no (e.g. children and prisoners) • Do you need informed consent for all those in treatment communities when not everyone is surveyed? • Do people opt into the program or is entire community impacted? • Sometimes ask consent at community meeting • Data that could identify an individual must be kept confidential • As early as possible take out information that could identify individuals • Use deidentified data for analysis and publication

11. Justice provision • Test questions of relevance to those involved in the study • Unethical to test a drug on prisoners and only sell drug to rich people • Best if participants themselves gain from findings • E.g. the program found to work, scaled up in study location • Not always possible • Ethical to test program designed to help poor Ethiopian farmers on poor Ethiopian farmers

12. Beneficeprinciple • Potential benefits must outweigh potential harm • Researcher should seek to minimize risk • Is researcher responsible for risk of program or only for the risk of harm associated with the evaluation? • Program may have risks but that does not make evaluating it unethical • Are program participants informed of program risks? • Would the program have gone ahead anyway? • Researchers should seek to minimize risk in program they evaluate but important to evaluate risky programs that are going ahead anyway • Angrist, 1990 evaluated impact of draft lottery for Vietnam War

13. Risk of harm from evaluating • Do fewer people receive the program because of the evaluation? • Ethical to reduce coverage if benefits of program unclear and evaluation can improve effectiveness of program • Do different people receive the program? • Is the program less well targeted than it would have been without the evaluation? • Cost of less good targeting needs to be offset against gains from evaluation • Will evaluation help improve knowledge of who to target? • Risk of confidential data becoming public • See respect for persons

14. What are the benefits? • If we answer an important question well, this can have big benefits to society and those studied • If we find harm, program can be shut down • If we find benefit, program can be extended • Learn lessons potentially relevant to other situations • The ability of randomized evaluations to learn about causality is relevant for ethics • The better we answer the question the higher the benefit, key component of ethics considerations