Vinod Thomas, Director-General Independent Evaluation Group

1. ECA: New Frontiers in Achieving Results* Plenary April 1, 2008, Improving the Lives of People in ECA Vinod Thomas, Director-General Independent Evaluation Group *This presentation is based on IEG’s recent evaluation findings. Ann Flanagan and Jiro Tominaga contributed to its preparation.

2. Progress and shortfalls in world development • Per capita economic growth averaged 2% word-wide, 4% in developing countries and 6% in ECA (2000-06) • Income levels diverged across countries, but converged across populations (due to rising income in populous China and India) • Social indicators – infant mortality, school enrollment – tended to converge worldwide since 1990 • The proportion of the poor declined since 1990, but the absolute number of poor is still around 1 billion • Environmental degradation continued to rise everywhere -- especially pollution, land degradation, deforestation • Progress in corruption has been slow in most countries, while stronger in ECA (1999-2006)

3. Growth, distribution and poverty

4. Development Effectiveness Local Policymakers Country Outcomes Policies Political Economy Knowledge Attribution of results is not easy Private Capital Flows International Financial Institutions

5. I. Connecting operations to results Quality of People’s Lives • ECA Projects & AAA • Moldova: AIDS Control • Turkey: Earthquake • Romania: Rural Educ • Bulgaria: Rev. Admin • Kyrgyz: Social Prot • Ukraine: People’s Voice • Etc. Project-Program Equity Efficiency Sustainability Public-Private Knowledge- Lending Country-Global → → →2008 → → → → →2015→ → → → → 2030 → → → Time

6. 1. WB projects outcomes improving Source: Staff calculations, IEG Database

7. …. but with program disconnects World Bank ECA Source: Staff calculations based on IEG project ratings (IEG Database) and IEG CAE ratings (1985-2006)

8. 2. Private sector can complement outcomes Development outcome andIFC’s profitability (ECA) Additionality and project development results (ECA) Note: ECA region by number of projects based on 178 projects from 1996-2007 XPSRs.

9. Need public-private coordination for better results IEG reports document crucial gaps in WB-IFC collaboration, but there are examples of linkages: • Leasing: IFC leasing projects and technical assistance contributed to the Bank’s rural agriculture leasing projects in Azerbaijan,Kazakhstan, Ukraine • BTC pipeline: In Azerbaijan the Bank focused on transparency and fiscal management while IFC concentrated on mobilizing consortium, social and environmental sustainability and SME linkages. • Sub-national finance: The Bank maintained relationships with regional governments. IFC added value with its ability to finance without a sovereign guarantee.

10. 3. Knowledge, lending and transition • AAA rated as moderately effective or better in influencing government decisions by 2/3 of the MIC report client respondents: e.g., Ukraine’s WTO accession process • Integration with country strategies: knowledge services provided entry for relationship with the government on public sector reform • Knowledge-led strategies: QAG ratings of AAA programs in 8 countries with high budgets for knowledge lower than those with lower shares of AAA • Difficult to use AAA on its own as a strategic instrument • Some countries to agree on cost-sharing for AAA based on multi-year program – Kazakhstan JERP • IEG’s forthcoming AAA evaluation: 12 cases including Romania and Serbia. • Role in informing Bank strategy? Link to lending? When AAA can be decoupled from lending? Adequacy of results framework and follow up in AAA? Source: IEG, Development Results in Middle-Income Countries; and Evaluation of World Bank Support for Public Sector Reform

11. Lending & policy: PSR lending, CPIA 99-06 • ECA has the highest rate of CPIA governance score improvements for countries getting PSR lending • The rate of improvement for non-borrowers is almost as high • External factors e.g. EU accession may be in play Note: Total number of countries in parenthesis Source: WB CPIA Scores and IEG staff calculations

12. 4. Regional/global and country programs • With the growing importance of trans-border issues, regional programs (such as building swifter trade routes Southeast Europe) and global programs (such as GEF) have vast potential • At $3.5 billion in 2006, regional programs account for less than 1 percent of the Bank’s total project and partnership financing. • Effectiveness hampered by lack of connection between country programs and regional/global initiatives. • M&E especially weak in regional and global programs Source: IEG, Annual Report on Operations Evaluation 2004 and 2005.

13. II Role of M&E in the results agenda Source: IEG Database

14. Results chain and program effectiveness A well articulated results chain would (ARDE 2006): • Results-based CASs, if done well, puts the focus on how projects add up to country impact • Emphasize realism in setting objectives – a problem in 50% of Country Assistance Strategies (FY2001-FY2005) evaluated by IEG • Track key synergies across sectors – for example, infrastructure for education results or environment for health results • Focus on capacity building – after all, country M&E determine the application of a results chain in countries

15. ECA’s experience in M&E • Performance-based budgeting of various types being followed in Czech Republic, Kazakhstan, Russia, Slovenia and Turkey • Especially EU accession countries are following OECD standards; some are becoming donors -- Czech Republic, Latvia, Poland and the Russian Federation. Greater accountability requires M&E capacity • Projects: The South East European trade and transport facilitation project introduced electronic forms showing duties due. Turkey brought extra-budgetary funds (that had undermined fiscal discipline) into the budget and parliamentary scrutiny

16. Impact evaluation can strengthen M&E • IE can promote or question programs – a crucial role of knowledge • Found the T&V extension system used in Kenyan agriculture to have no impact on agricultural production • It tracks linkages across sectors – including unintended and indirect • Showed the linkage between a nutrition program in Bangladesh and secondary schooling for girls education • It de-politicizes decision-making – contributes to sustainability • Evidence-based case for the Mexican Opportunidades program made it difficult for politicians to discontinue support for after change of government • Initiatives at the Bank: Development Impact Evaluation Initiative; IFC Advisory Services Impact Evaluation Initiative; Spanish World Bank Trust Fund for Impact Evaluation; African Impact Evaluation Initiative; IEG Impact Evaluation Program • Challenges ahead: be demand-driven; enter programs ex ante; be theory based; focus IE on the most relevant issues

17. ECA lags in impact evaluation Source: IEG staff calculations (draft), DIME Database and Poverty Impact Evaluations Database Note: The Poverty Impact Evaluations Database contains evaluations conducted by both Bank and Non-Bank staff.

18. Four cutting edge challenges in ECA • Project-program: Especially where Bank lending share is small, there is a premium on exploiting linkages across projects and sectors for the greatest country impact • Public-private: It is vital to develop public-private partnerships and to make the most of public and private linkages • Knowledge-lending: The knowledge-lending axis is exploited differently across countries • Country-global: The country-based approach must be reformed to integrate tough issues of climate change, governance, migration…

19. Thank You