What is the Evidence of the Impact of Microfinance on the Well-being of Poor People?

1. What is the Evidence of the Impact of Microfinance on the Well-being of Poor People? Maren Duvendack1, 3 Richard Palmer-Jones1 With J. Copestake2, L. Hooper1, Y. Loke1, N. Rao1 3ie/LIDC, 29 June 2011 Funded by 1. UEA, 2. University of Bath 3. IFPRI

2. Introduction • Objectives: • Assess impact of microfinance on social and economic well-being of people living in developing countries and are poor, excluded or marginalised within their own society. • Methods: • Adapted to social science studies from Cochrane and Campbell Collaboration and EPPI-centre guidelines • Results: • 2 RCTs, 9 pipeline studies, remainder w/wo studies. • No robust evidence on most economic, social and empowerment outcomes. • Negative as well as positive impacts. • Many studies based on weak research designs and problematic analysis.

3. Inclusion Criteria • Participants: • in poor, lower and upper-middle income countries • Intervention: • credit, credit plus and credit plus plus • Comparison group: • control group w/o microcredit • Outcomes: • economic, social and empowerment outcomes • Cut-off point: • studies published since 1970 • Methodologies: • RCTs, pipelines, before/after & with/without studies, >100 cases • Publication status: • formal and informal  74 papers examining approx 20 broader economic, social and empowerment outcomes

4. Search Strategy Search strategy Records identified through database(11) searching n=3,620 Additional record identified through other sources (12) n= 115 Records after duplicates removed n=2,643 Screening Recordsscreened (abstracts and titles) n=2,643 Records excluded n=2,442 Eligibility Full-text articles assessed for eligibility n = 201 Full-text articles excluded, with reasons n = 127 Included Papers included in synthesis n = 74 Further screening -> n = 58/29 studies

5. Data Extraction & Validity Assessment • Assessment of validity focused on: • Assessing the intervention & measurement of outcomes • Contextual factors affecting outcome heterogeneity (sub-group analysis) • Research design & analytical method • Assessment took a long time because • Abstractsnot well structured or methodologically informative • Few studies used rigorous research designs • RCTs, pipeline, with/without, natural experiments • Most/all papers suffered weak external and/or internal validity • Reliance on sophisticated statistical analysis to obtain impact estimates • Need for replication because of • Data processing or computational errors, • Alternative analytical methods and/or assumptions etc.

6. Therefore select studies based on scoring • Selection based on Hierarchy of methods • Weak research design requires more sophisticated methods of analysis to attain similar levels of validity • High validity Low validity RCTs, pipelines; with and without; natural experiments2SLS, instrumental variables, PSM, DiD, control function • Combined score into an index with fuzzy cut-off point • Arbitrary but transparent approach reduced 74 papers 58 papers • Strict inclusion criteria  too few studies meet orthodox validity criteria • but relaxing inclusion criteria  too many included!

7. Results – Research Designs • 2 RCTs – several problems – not gold standard • Weak external and internal validity • Banerjee et al., 2009/10; Karlan and Zinman, 2010 • Pipelines • Are present and future participants good matches? • Drop-outs, graduates, contextual factors, small samples • With and without – panel and cross section • High risk of bias • Requires complex statistics of questionable validity • Doubts about data-mining, and replicability of analyses • Need to compare with alternative assumptions about causation You cannot substitute weak research design (and poor data) with complex statistics (Meyer and Fienberg, 1992)

8. Results – Number of Outcomes • Most outcomes tested are early in the causal chain • Huge number of tests – 58 papers 2869 impact estimates

9. Results – Impact Signs and Significances • More estimates non-significant than significant • Majority of +ive impacts, but most not statistically significant • Many negative impacts (significant and non-significant) • Multiple testing – many estimates using same data • Failure to adjust test statistics (many use 10% level) • Ethical research, analysis and dissemination • No evidence on missing studies, or analyses

10. Conclusion • Common belief is that microfinance is pro-poor and pro-women • BUT: little convincing evidence • Almost all MF IEs have high vulnerability to bias • Most worryingly RCTs which are the best regarded/widely quoted studies • Suffer from weak methodologies and inadequate data which adversely affects impact estimates • Unclear under what circumstances and for whom MF works • Recommendations • Focus on need for more and better research • Conduct well designed quasi-experimental and observational studies including longitudinal studies, earlier in fashion cycle • Replication of highly regarded studies of whatever research design • Capacity building in multi-disciplinary & mixed methods research

11. THANK YOU!

12. Causal Pathways

13. Results – Distribution of Methodologies * 2 papers (Chen and Snodgrass, 1999 and Dunn, 1999) are included in our analysis but are missing from this table since they had a high score (2 and above). We included them in our synthesis because they were part of a group of papers that used the same data set, i.e. the USAID data on India and Peru.