Power and Pitfalls of Experiments in Development Economics: Some Non‐random Reflections

1. Power and Pitfalls of Experiments in Development Economics: Some Non‐random Reflections Christopher B. Barrett Michael R. Carter Presented at the annual meetings of the Agricultural and Applied Economics Association (AAEA)July 2010Denver, Colorado

2. Overview Experiments now wildly popular in economics overall. So what role do they play in development economics? Subject of considerable dispute currently (Deaton and Heckman vs. Banerjee, Duflo and Imbens) We study human beings as agents whose choices, conditioned by the external environment, result in behaviors that matter not just to their own well-being but also, due to externalities and general equilibrium effects, to the aggregate experience of their communities.

3. Overview Ultimately development economists interested in welfare and behavioral outcomes:  y=f(s,p,b,e,α,ε) and b= g(s,p,y,e,θ,φ) s = structural variables p = policy variables e = elicitable behavioral characteristics α/θ = unobservable characteristics Estimation challenges: (i) simultaneity of b and y, (ii) unknown true functional form (iii) the common non-orthogonality of (α+ε) and the b,p,s and e variables of primary interest.

4. Two different threads of experimental work to address these challenges: • Behavioral field experiments to elicit e or to identify ∂b/∂p cleanly. 2) Randomized controlled trials (or social experiments) aim to identify ∂y/∂p cleanly.

5. 2. The Power of Experiments: Behavioral field experiments … • Substantially reduce unobserved heterogeneity and endogeneity concerns. Ex: key risk, time and trust preference parameters. Carter and Castillo (2005) study of Hurricane Mitch in Honduras. 2) Can replicate realistic choice settings to test key hypotheses. Ex: BDM auctions to identify effectiveness of free distribution of insecticide treated bednets in Uganda (Hoffmann et al. 2009). … remain underemployed in development economics.

6. 2. The Power of Experiments: RCTs… in contrast to BE, these are now dominant in empirical dev’t economics • May resolve econometric problems associated with program placement and selection effects as well as the endogeneity of key p or s variables. 2) Use random assignment to evaluate pilots and do more rigorous program evaluations. Huge donor demand for more evidence of near-term impacts.

7. 2. The Pitfalls of Experiments: A) Internal Validity The supposed trump claim of experiments, esp. RCTs, often not as strong as claimed. • Randomization bias • Faux exogeneity where true treatment effects are subjective and unmeasurable • Ideal designs commonly compromised in field implementation, esp. by non-research partners

8. 2. The Pitfalls of Experiments: B) External Validity Unobservable and observable features inevitably vary at community level and cannot be controlled for in experimental design because context matters. • Non-random implementation partner • Essential heterogeneity – experimental results are unknown weighted average of heterogeneous responses … may be no external population that matches the sample mean from the RCT!

9. 2. The Pitfalls of Experiments: C) Asking The Right Questions? Only a few relevant topics are amenable to randomization. Omits macro, political, GE, negative shocks questions. • Limited ability to study phenomena with lagged effects (e.g., early childhood interventions) • RCTs only identify the mean treatment effect. Often want other characteristics of dist’n of effects: conditional effects, proportion +/-, etc. • Crucial distinction between efficacy (the study of a treatment’s capacity to have an effect under controlled conditions) and effectiveness (real world impact). Overcorrection for endogeneity can render findings consistently and unbiasedly irrelevant. Serious risks of distorting research agendas … economics reduced to evaluation … often of points obvious to laypeople.

10. 2. The Pitfalls of Experiments: D) What Opportunity Cost? • Behavioral experiments aim to identify and explain variation in behaviors. Good. • RCTs compare against the “no intervention” counterfactual. Infeasible to do multi-factorial randomized block design of high order dimensionality. Leads to distorted recommendations (example: deworming).

11. 2. The Pitfalls of Experiments: E) Ethical Concerns Class 1: Predictably violate basic “do no harm” obligation. IRBs not sufficient. Example: Bertrand et al. (2007 QJE) Class 2: Ignore responsibility to secure informed consent. RCTs commonly blind subjects to intervention to avoid endogenous behavioral responses. Class 3: Suspend targeting principle, compromising the expected effectiveness of resources expended in addressing social ills. Explicitly ignore local information.

12. 3. Access to Capital as Central Issue in Development Economics • Evolution of capital access theory and policy • Monopolistic perspective  public banking & interest rate regulation • Laissez faire in theory & practice • Imperfect information & the search for collateral substitutes • Evaluation of credit markets & interventions is difficult • Double selection in credit markets heightens concerns over separating the impact of capital access from the impact of the characteristics of those with credit • Fundamental Identification problem of (potentially) disequilibrium market: Observed transactions (loan or no loan) do not allow complete sorting of observations into correct behavioral regime:

13. 3. Access to Capital as Central Issue in Development Economics • Key empirical questions regarding capital access: • Does the financial market work such that we find households in the non-price rationed regimes or do markets work in an efficient, price-rationed manner for all? • If there is non-price rationing, is it systematically biased against any particular set of households (e.g., low wealth households) such that the operation of the competitive economy tends to reinforce initial levels of poverty and inequality? • How costly is non-price rationing and how much would household input use and income increase if liquidity constraints could be relaxed and non-price rationing eliminated? • Are there contractual or institutional innovations that can change the rules of access to capital, lessen non-price rationing and decrease its cost? • Econometric analysis of observational data has long struggled to answer these questions: • Control for latent characteristics (panel and Heckmanesque methods) • Getting the regime sorting (regression heterogeneity) right by getting beyond naïve approaches • RCT and behavioral experiments seem potentially valuable—indeed success claimed for RCTs in realm of credit.

14. RCTs & Existence of Credit Rationing • Theory suggests that under asymmetric information, interest rate increases can cause lenders to decline because these price increases induce adverse selection & moral hazard. Would therefore expect lenders to self-impose interest rate ceilings, resulting in excess demand at that rate & market clearing via quantity rationing. • If correct, this would imply that higher interest rates should result in more default and lower profit via either adverse selection or moral hazard. • In an innovative experiment, Karlan & Zinman (Econometrica, 2010) worked with a South African paycheck lender to randomize the interest rate to see if higher rates had this impact • Existing clients invited to borrow at an announced rate • Some were then offered a lower rate when they applied • Find very little evidence that higher rates influence default

15. RCTs & Existence of Credit Rationing • At best this strategy would only take us part way towards understanding the existence and incidence of non-price rationing as it only operated on an existing base of borrowers and would say nothing about those that already rationed out • While no study can do it all, there is another level at which we may wonder about the meaning of such a price variation experiment: • In a companion paper, Karlan & Zinman (AER 2008) use this data to the price elasticity of demand for credit • Find a kink in the relationship around the ‘normal’ market price: demand responds to prices above that price, but does not respond to prices below that price • One explanation could be that the some of the treated did not find the announced lower price credible (there is no free lunch!) • From this perspective, we have to ask if we can really randomize things like prices that have a social meaning and depend on the subject’s perceptions. Was this a case of faux randomization where the true treatment received was endogenously determined by the subject’s education and sophistication?

16. Randomized Liquidity Injections 1: Santa Claus Treatment Effects Normal credit selection processes inexorably make access to capital correlated with hard to observe borrower characteristics, leading to identification problems De Mel, MacKenzie & Woodruff (QJE, 2009) suggest solving this problem by simply randomly distributing liquidity injections to small scale entrepreneurs in Sri Lanka. While solves one of the identification problems noted above, it does not deal with the regime switching which has been a central preoccupation of the empirical literature for 20 years. Similar to some of the older naïve econometric literature, this approach only identifies the impact of capital under the assumption that all businesses were in an excess demand regime If not true, then estimated treatment effects are an unknown, data-weighted average of multiple regression regimes From a policy perspective, does not tell us what to expect if credit access were expanded through normal banking, as opposed to ‘Santa Claus’ mechanisms

17. Randomized Liquidity Injections: Take 2 • One seeming way to circumvent this problem of the de Mel et al. study would be to give liquidity injections only to those who reveal excess demand by applying for loans • Working with the same paycheck lender mentioned above, Karlan and Zinman (Rev of Fin Studies, 2009) created a design to de-ration some randomly selected applicants whose credit scores deemed them credit unworthy • However, because this study was creating real debt (unlike Santa Claus liquidity gifts), several problems ensued: • Loan officers failed to comply with random assignment almost 50% of the time • Experiment exposed borrowers to real harm if they could not repay the debt—a non-trivial concern given that the lender’s scoring model in fact predicted repayment problems for these de-rationed agents • Under standard human subjects protection protocols, experiments like this would require full disclosure (no deception) and procedures to compensate for any harm caused by the treatment • However, implementing such standard protections would likely invalidate the results of the study (telling borrowers that their experimentally induced debt would be repaid in case of default would clearly alter repayment incentives and behavior) • Again see how economic experimentation is very different than, say, drug trials

18. Purging the Error Term with Behavioral Experiments While the RCTs just discussed try to break the problematic correlation between error term and capital access by randomizing the latter, an alternative approach is use behavioral experiments to make unobservables like risk aversion, business acumen and time preferences observable and thereby purge the error term of its most problematic part Binswanger’s early and influential field experiments on risk aversion (EJ, 1980; AJAE, 1981) suggest that this can be done (though his own results were problematic) A few studies are moving in this direction, but still much to be done on credible measurement of these kinds of characteristics

19. Changing the Structural Determinants of Capital Access • Theory offers a number of insights into the structural conditions (risk, asymmetric information, etc.) that create non-price rationing • An ambitious approach seen in some recent studies is to innovate and implement new contracts and institutions designed to change these fundamentals and induce new rules of credit access: • McIntosh, de Janvry & Sadoulet (EJ 2010) on credit bureaus in Guatemala • Gine and Yang (JDE?)on biometrics in credit markets in Malawi • I4 Index Insurance Innovation Initiative projects in Peru (Boucher, Trivelli & Carter), Ethiopia (McIntosh, Sarris & Ahmad) and Mali (Bellemer, Guirkinger & Carter) • All of these rely on real institutions (hard!) and spatially randomized rollout • Stay tuned!

20. Conclusions • Challenge for contemporary development economics is to keep its balance of rigor and relevance: • Experiments rightly play a powerful role in modern development economics in complementing theory and observational data to understand underlying structure and enable descriptive, predictive and prescriptive analysis. • Behavioral experiments, in particular, can play a still-greater role in teasing out credible estimates of otherwise-unobservable parameters that matter to estimating behavioral and welfare response. • Some rollback of the RCT obsession would be welcome. There is no ‘gold standard’ of perfect identification, and we need to beware of the blind pursuit of exogenous variation lest it crucify development economics on a cross of golden irrelevance.

21. Thank you for your comments