Evaluating social safety nets at the World Bank: Country case studies –Jamaica

1. Evaluating social safety nets at the World Bank: Country case studies –Jamaica AEA meetings 2010 Victoria Monchuk IEG World Bank

2. Evaluation building blocks • 30 country case studies: looking beyond portfolio • 25 randomly selected desk- and interview-based • 5 purposely selected field-based (Indonesia, Ethiopia, Colombia, Jamaica, Georgia) • 3 in-depth Project Performance Assessment Reviews (projects in Ethiopia, Colombia, Jamaica) Examine the relevance and nature of the Bank’s engagement; country’s application of social risk management framework (SP Strategy, 2001); and the results in countries

3. 30 cases: in-depth and broad dataset In–depth: Case studies serve as in-depth understanding of how Bank’s approaches played out in a country context, explanations for the “why”, examples to illustrate points, anecdotes, etc. Breadth: Data set (n=30) of qualitative data and data that can be quantified to see trends across regions, low- and middle-income countries, etc.

4. Jamaica context • 2nd poorest in Caribbean after Haiti • Decades of slow growth (ave. 1.5%) • Sensitive to shocks • Poverty has declined, but deep pockets remain • 49% of the poor are children • Unemployment high and strongly related to poverty • Debt burden severely restricts social spending (144% of GDP, interest payments of 64% of GDP in 2001)

5. Safety nets in Jamaica • Safety net consists of: • Transfers: food stamps, cash and in-kind to elderly/indigent • Community-based programs: Jamaica Social Investment Fund • School/health-based programs: school feeding, drugs for elderly • 2000 SSN reform: consolidate several welfare programs for families and elderly into one Conditional Cash Transfer program (discontinue food stamps) • 2001 Program for the Advancement through Health and Education (PATH) launched

6. Bank support to Safety Nets 2000-10 PATH CCT Analytical and technical assistance SSN lending: Social Safety Net project (2001-09) Social Protection project (2009-13) Social Investment Fund project (1996-2001) National Community Development project (2002-08) JSIF Hurricane Dean Emergency Recovery Loan (2007-11) Development policy loan (2010)

7. Look at projects and beyond • Ex-post project evaluations assesses relevance and effectiveness of projects against their objectives • Does not look much beyond the project • Case studies assess the effectiveness of the whole SSN system and of the entirety of Bank support • Relies on project performance and impact evaluations for outcomes and impacts

8. Case study design Context (economic, political, poverty) Country’s objectives and SSN systems Bank’s objectives and SSN support Assessment of relevance and effectiveness of Bank-supported SSNs Assessment of the Bank’s engagement in developing SSNs Ratings Hypotheses

9. Approaches • Literature review • Interviews with relevant Bank and IDB staff • Country visit: • Interviews with Bank staff • Interviews with government clients (Planning Institute, Social Fund, MLSS) and data collection • Field visits and interviews with parish office staff, school and health center staff, beneficiaries • Interviews/focus group with development partners (IDB, DfID, EU, UNDP, CIDA) • Follow-up interviews with Bank and IDB staff

11. Project findings Social Safety Net project achieved its objectives of strengthening the SSN system, providing more cost-effectives social assistance, and targeting the poor and vulnerable Small impact on human capital investment and poverty Although not designed as such, the PATH program has been able to provide support in response to shocks

12. Project findings National Community Development project increased the access of poor communities to good quality basic services and roads and created some temporary jobs Modestly increased social and community empowerment Did not incorporate mechanisms for ensuring unskilled employment, did not measure impact of employment on poor households

13. Findings on Jamaica’s SSN • Needs stronger emphasis on creating income and job opportunities and links to labor markets • No sustainable mechanisms for responding to systemic shocks • FFF response: scale up coverage (47%) and benefit levels • FFF response: universal health and sec. school fees waivers • Because fiscal room is minimal, it is important to put in place a safety net that can be scaled up in times of crisis and down in more stable times • Developing a social protection strategy only in 2009

14. Case studies link to other evaluation building blocks • Portfolio review gave clues that it was important to distinguish between instruments and design details • Cases provided explanations for choices made in the country leading to an in-depth review of results frameworks in SSN projects • FFF survey exposed Bank readiness to help countries respond to the crisis through SSN systems • Cases showed how individual countries used available resources • Public works review provided evidence on strengths and weaknesses • Cases provided context for Public Works projects and other programs

15. Main take-aways: Ex-post project evaluations give specific information about project achievements and why Case studies help answer questions on the country and sector level, the effects on whole SSN systems, and the Bank’s strategic relevance Case study data triangulate with other sources Project evaluations: worldbank.org/ieg