SOME PRACTICAL ISSUES IN THE IMPLEMENTATION OF RCTs

1. SOME PRACTICAL ISSUES IN THE IMPLEMENTATION OF RCTs Martín Valdivia

2. Idea of the workshop • Transmit some experience about the implementation of RCTs • Own work (more failures than successes) • Picked up at workshops, seminars • Literature review • As a base to start up discussion • At the 3 key stages • Design • Monitoring • Analysis

3. Designing issues: The research questions • Relevance of RCT with one treatment • Allows to establish if a particular intervention has a positive impact, • Also useful to establish relative importance (cost-benefit analysis) • Some limitations • Not that informative when intervention includes many components (PROGRESA case, are the conditions necessary?, which of the components makes for most of the differences?) • Threatening for program officers if taken as an evaluation of their work

4. Designing issues: The research questions • Multiple treatments may help • Disentangle the individual contributions of each component of the intervention • Resources vs. autonomy for schools in the Kenyan case • Easier to argue that the evaluation strategy can be considered a management tool • Example 1: expanding primary health services with two alternative methods • contracting private providers vs • adjusting incentives for regular public providers • Some key considerations • Question needs to be valid for key decision makers (one option cannot be dominated by the other in the minds of decision makers) – design will not be accepted • Implementing agents need to be convinced to do their best in each option – danger of sabotage by agents

5. Designing issues: The research questions • RCTs can also work with ongoing programs, being easier to present as a management tool • Group vs. individual liability (Karlan & Giné, 2007) • Group liability or just peer pressure. Switch to individual liability did not worsen repayment, increased outreach • Business training for microfinance clients (Karlan & Valdivia) • We can teach entrepreneurship, but concentration of benefits among the initially less interested has implications for delivery

6. Designing issues: Incorporating the perspective of the implementing agency • Often times, refusal to RCTs is based on the fact that practitioners want to maximize the probability of success • They want to use their knowledge about: • Who needs the program most • Who are the most likely to benefit the most • However, this can be incorporated into the RCT design • Ask them to use that information to determine an eligible group • But, just make sure their eligible group is larger than the initial implementing capacities • The Peruvian rural roads example • Obviously, this affects the external validity of the results • Impacts will not be shared when beneficiaries include less eligible individuals or communities

7. Designing RCTs: Ethical issues • Justifications for a control group • We do not know if program will be effective • However, program officials often do not work that way. They tend to move in favor of an innovation when convinced of the benefits of the program • Useful to think in terms of phasing in the implementation based on limited capacities at the beginning (financial and institutional capabilities) • Then, the only decision is who becomes a beneficiary at the beginning • Randomization may be the fair way to decide among eligible beneficiaries

8. Designing RCTs: The Political Economy of RCTs • It may be useful to analyze the differentiated interests across key actors involved • A Peruvian case in microfinance: FINCA vs. USAID • Ministries of Education trying to convince Ministries of Finance may work • Then, partnerships with researchers become useful for implementing agencies

9. Designing RCTs: Take-up • CCT studies have tended to omit discussions on take-up • Although, vastly discussed in the labor economics literature, around the training programs • You randomize offering treatment, take up is endogenous • Including all those eligible in the analysis (baseline and follow up surveys) may allow for • Analyzing take up • Analyzing ITT and TOT • Very useful information for future scaling up decisions

10. Monitoring • Tension between program officials and experimental design often does not end with initial agreement • Important to establish a system to collect timely information about the way the program is implemented • Identify in time threats to the experimental design • Identify mechanisms through which effects are appearing, or why not • Examples: • Program periodic meetings with program officials to discuss the way implementation is going • A “spy” may should not be discarded • Focus groups with program participants • Collect data on attendance, dropouts, etc.

11. Analysis: Multiples Outcomes • It is important to report estimated impacts for all outcomes • However, looking at too many outcomes independently increases the probability of false rejection of null hypothesis • Kling, Jeffrey; Jeffrey Liebman; Lawrence Katz (2007). "Experimental Analysis of Neighborhood Effects". Econometrica 75 (1): 83-119, January • We can account for correlation among outcomes of a family • Transform outcomes so that “more is better” • Normalize outcomes by using mean and sd of control group • Sum normalized outcomes of a family and evaluate DD coefficient • run a SUR model and test significance of the sum of the individual DD coefficients of a family

12. Analysis: Attrition • Random attrition is mostly harmless • Although, it does affect the statistical power of your analysis • Non-random attrition might bias our estimates • If dropout is larger among those that benefit the most, ignoring attrition leads towards underestimated the program's effects • First, do your best to avoid attrition • Robustness analysis will include parametric and non-parametric approaches • Hausman – Wise selection model • Inverse probability weighting • Wooldridge, Jeffrey (2002): “Inverse Probability Weighted M-Estimators for Sample Selection, Attrition, and Stratification". Portuguese Economic Journal, 1: 117-139.

13. Analysis: Attrition • Bounds analysis (non-parametric) • Worst-case bounds (Horowitz & Manski, 1998) • With bounded outcomes, upper (lower) bound is established by assuming the best (worst) outcome for those attrited • Horowitz, Joel L.; Charles F. Manski (2000). "Nonparametric Analysis of Randomized Experiments with Missing Covariate and Outcome Data". Journal of the American Statistical Association 95 (449): 77-84, March. • Trimming • Using theory to reduce the range • Lee, David S. (2002). "Trimming for Bounds on Treatment Effects with Missing Outcomes". NBER Technical Working Paper # 277, June.