Evaluating Social Funds: A Cross-Country Analysis of Community Investments

1. Evaluating Social Funds: A Cross-Country Analysis of Community Investments February 2004 Laura Rawlings, Lynne Sherburne-Benz, Julie Van Domelen

2. Context for the Evaluation • What are Social Funds? • Financing mechanisms that promote public investments in small projects identified and carried out by local groups (communities, local governments and NGOs) • Why are they important? • Social funds have received US$8 billion in financing from multilateral development banks, bilateral donors, and government/ community contributions, including US$3.5 billion World Bank financing for 98 projects in 58 countries • First large-scale experience of the World Bank in financing small projects from communities as opposed to top-down investment plans from sector ministries

3. Objectives of the Evaluation • Gaps in knowledge • Are social funds successful at reaching the poor? • Do these investments translate into improvements in well-being? • Are the impacts sustainable? • How cost-effective are social funds compared to other mechanisms? • Objectives of this study • For the first time, utilize primary data, including extensive household-level surveys, to assess: • Poverty Targeting • Benefits • Sustainability • Cost

4. Elements of the Evaluation • Country Case Studies • Armenia, Bolivia, Honduras, Nicaragua, Peru, Zambia • Focus on Social Infrastructure Investments • Broad Participation and Capacity Building • World Bank staff, social fund staff, researchers, and often, national statistics agencies • Donors: Various support to country case studies and Norwegian Trust Fund to initial design of cross regional and supplemental data efforts

5. Evaluation Methodology • Impact Evaluation -Based on the construction of a counterfactual using a variety of methods -Quality and sustainability of facilities and infrastructure; coverage and utilization; household-level health and education status • Targeting -Relative geographical allocation to districts or municipalities -Benefit-incidence analysis of whether social funds reach poor households • Costs, efficiency relative to other investment mechanisms • Priorities and Participation

6. Findings on Geographical Targeting  Geographic distribution of resources is variably progressive -- poorer districts receive more per capita

7. Findings on Geographical Targeting • Social fund geographical targeting has improved over time in all cases, due to: • Evolution of social fund goals - away from emergency • Proactive outreach (regional offices, info campaigns) • Positive discrimination in $ allocation using Poverty Maps • Increase in demand responsiveness - demonstration effect on more remote communities • Exogenous factors - e.g. fiscal decentralization • Large variation in per capita allocations between districts - not correlated with poverty level

8. Findings on Household Targeting-analysis of income/consumption deciles • Social funds tend to have either a neutral or mildly pro-poor distribution of beneficiaries

9. Findings on Household Targeting-Analysis based on income/consumption deciles • Poorest of the poor are reached • Bottom decile accounts for 8-15% • But also leakage to the better-off • Top decile accounts for 2-11% • Concentration of benefits among the poor varies by type of project • Latrines and health centers most pro-poor • Education and water also pro-poor • Sewerage regressive

10. Comparison with Other Programs- Geographic Distribution of Resources • Across regions, social fund allocations more pro-poor than comparators, but data only for a few cases: • Bolivia - SF expenditures 3 times more likely to reach a poor municipality than fiscal transfers • Peru - FONCODES expenditures better targeted to poor districts than other national social programs (PRONAA, INFES) • Across households, social funds performed at least as well if not better than other programs • Peru – best of 3 school programs • Armenia – comparable to other national safety net programs • Honduras - mid-range compared with 30 targeted social programs in Latin America (Grosh)

11. Impact Evaluation Results Education Social fund investments in education have ….. • a clear, positive impact on infrastructure • usually accompanied by an increase in the availability of non-infrastructure inputs (teachers, textbooks) • resulting in positive, significant changes in enrollment/attendance (Nicaragua, Peru, urban Zambia) and/or other outcome indicators (age for grade - Nicaragua, Honduras, rural Zambia; dropout - Bolivia) • No impact on achievement in Bolivia Health Social fund investments in health have ….. • a clear, positive impact on infrastructure • usually accompanied by an increase in the availability of key non-infrastructure inputs (medical equipment, staff) • although levels of key inputs remain low in SF and non-SF facilities • the generally positive, significant impact on utilization is not accompanied by clear, significant changes in health outcomes, except in Bolivia

12. Impact of Social Fund on Health Outcomes in Bolivia (Resto Rural and Chaco combined) Social Fund Mean Propensity Match Control Mean 1993 1997 1993 1997 Infant Mortality Rate 61.5 30.8 59.8 67.2 Under 5 Mortality Rate 94.0 54.6 92.6 107.9 • In Bolivia, the social fund investments significantly reduced infant and child mortality rates  Bolivia was the only case study designed to capture impacts from health investments on mortality rates

13. Water and Sanitation Social fund investments in water have ….. • increased access to water, raising coverage and reducing time and distance to the nearest water source • resulting in positive, significant changes in health outcomes in three of the four cases studied (all but Honduras), including <5 child mortality impacts --Bolivia: impact on duration of diarrhea; impact on under 5 child mortality of water and sanitation investments (reduction of 40% from 105 to 61 per 1000) --Peru -- Child mortality was reduced as a result of these improved health outcomes. Deaths of children under 5 were 33 per 1000 for social fund households benefiting from a water project versus 60 per 1000 in control group communities.

14. Social fund investments in sewerage have… • increased access to sanitation, although connection rates remain low • resulting in positive impacts on health outcomes (diarrhea) in one of the two cases studied Social fund investments in latrines have… • increased access to sanitation • resutling in positive impacts on health outcomes in one of the two cases studied (Honduras, not Nicaragua)

15. Conclusions and Recommendations • Key design issues for social funds • Social funds will need to define more closely their poverty targeting expectations • Building in mechanisms for community participation and management have large pay-offs • Moving from emergency to longer-term requires continued adjustment of procedures and criteria, particular with respect to sectoral coordination with line ministries, poverty reduction policies • Recommendations for evaluations • It is essential to have program staff involved in the evaluation terms of reference and development of recommendations from impact evaluations; outsourcing the evaluation itself lends rigor and objectivity, especially for impact evaluations • Evaluations should be carried out across similar public sector programs to assess comparative effectiveness • Impact evaluations should be but one evaluation component

16. Conclusions and Recommendations • Lessons on the role of social funds within poverty alleviation strategies • Social funds have met the broad objectives they were designed to address • The limitations of social funds are also clear – they cannot substitute for effective sectoral policies and cannot fulfill all the investment financing needs of all poor communities