Knowledge-Based Country Programs An Evaluation of the World Bank Group Experience October 24, 2013

1. Knowledge-BasedCountry Programs An Evaluation of the World Bank Group Experience October 24, 2013

2. Knowledge activities of type evaluated here have more than doubled in past 10 years • Increasing client demand for TA and ESW (blue line). Country programs moving to more knowledge services content. • Account for about three quarters of all knowledge services. • For the Bank: core knowledge services about one-third of administrative budget in recent years compared to a quarter ten years ago.

3. Knowledge-based country programs, knowledge activities, and method • Rank based on four indicators of knowledge services use; selected country programs from high end of the ranking (KBCP’s). • Bulgaria, Chile, China, Kazakhstan, Kuwait, Malaysia, Russian Federation, South Africa, Thailand. • Knowledge activities selected linked to strategic priorities in CPS • Method builds on former Quality Assurance Group’s knowledge services assessment: • Relevance, technical quality, results, sustainability of results. • Desk reviews CPSs and projects; visits to countries

4. The Nine Countries Selected are Representative Intensive Bank Knowledge Users High-income (Kuwait) and upper-middle-income, including EU and OECD members (Bulgaria, Chile) Knowledge services program more than 50 percent of country services budget on average Diversified economic structure No or moderate Bank lending Fee-based knowledge services

5. Most Bank knowledge activities raised stakeholder awareness, some supported lending • Raise stakeholder awareness 75 percent • Design reform, help consensus, build capacity, support dialogue, 50 percent of activities • Supporting Bank strategy formulation or lending operations about a quarter of the activities

6. Good News on Outcome Achievement ofKnowledge Services About 80 percent of the knowledge activities reviewed were expected to have outcomes achieved or partly achieved No significant difference between TA and ESW Agriculture, education, and financial sector above average Private sector/economic policy & governance below average

7. Knowledge in support of lending had better results than self-standing knowledge services • Knowledge services in support of lending more likely to succeed • Bank presence in the country • More established relationships with counterparts • More likely to have better results framework, including M&E

8. More specifically, the ingredients of success • Activities conveying best practice, using local expertise, and providing actionable recommendations were more likely to succeed • 90 percent of successful conveyed international best practice and relevant examples (50 percent for unsuccessful) • 55 percent used local expertise (28 percent for unsuccessful) • 72 percent actionable recommendations (50 percent for unsuccessful) • Activities addressing recipient capacity had better outcomes • 52 percent developed/strengthened institutions (16 % for unsuccessful) • 71 percent strengthened analytical or policy capacity (20 % for unsuccessful) • 60 percent of Bank knowledge services contributed to developing or strengthening institutions

9. Areas of weakness • Less success when: • Unable to address issues relevant to the client (40 percent for unsuccessful activities) • No new evidence or data for decision making (30 percent for unsuccessful activities) • Activity not delivered on time for decisions (30 percent for unsuccessful activities) • Activities requiring coordination of reforms across sectors or new legislation (examples, private sector/economic policy & governance)

10. Monitoring and Evaluation a Significant Weakness of Knowledge Services • Monitoring of capacity development outcomes and lesson learning were, on average, weak for Bank knowledge services evaluated. • In only 17 percent of activities was there at least a partial results framework in the CPS. • And only 23 percent of knowledge services included, at least partly, result indicators to track the achievement of the activities’ outcomes.

11. The Evaluation Links to Change Agenda on Knowledge

12. A client focused engagement is essential → respond quickly to client requests, use programmatic approaches, tap local expertise. • Activities achieved outcomes more often if: • stayed engaged and responsive in implementation • translated recommendations into actions fitting local conditions → involve client early on in design and ensure that timing of delivery and follow up adequate to affect important decisions. • Challenges for the Bank: • More client participation in design • Better timing of delivery for decisions • More use of local expertise

13. Strategic focus and agile delivery of global expertise required to become the best development solutions provider → keep strategic focus, and ensure that delivery of knowledge services through a “consultant firm model” does not result in fragmentation of RAS programs • With shift from finance to knowledge: • stay engaged in areas relevant to clients’ development agenda (use programmatic knowledge services) • maintain capacity to see the big picture → deploy highly experienced staff with global perspective, able to formulate policy options that fit local conditions and generate data to support policy making. • Challenges for the Bank to enhance global practices: • Relevance and technical quality • Generate timely evidence or data to support policy making • Knowledge flow across organization/mediate knowledge from KBCP’s to LICs

14. Invest in monitoring and evaluation to learn and innovate → a significant investment in monitoring and evaluation will pay significant dividends for learning and for the effectiveness of knowledge services • Three ingredients of science of delivery—evidence-based design, agile implementation, structured learning—require strong M&E • In particular “rapid cycle learning” from project design to implementation requires a strengthened M&E framework for both projects and country programs

15. To Sum Up: • Country clients acknowledged Bank: • independent and credible knowledge broker—familiarity w/local context • capacity to customize international best practice solutions • ability to see the big picture and analyze cross-sectoral issues • Results could be improved by: • more client participation in the design of knowledge services • timing better the delivery of services to inform important policy decisions • making more use of local expertise • following up the knowledge services with policy implementation support where needed • In some instances Bank staff have insufficient experience/global expertise to deal with difficult policy problems: • deploying highly experienced staff with global perspective essential. • Improving monitoring and evaluation systems is an essential investment

16. Thank you.

17. Additional Slides These additional slides provide numbers supporting statements made in the presentation.

18. Successful Knowledge Activities Activities conveying best practice, using local expertise, and providing actionable recommendations were more likely to succeed

19. Less Successful Knowledge Activities Activities that were unable to address issues relevant to the client, with little new data to inform decisions, or not delivered in time for decisions were associated with less success in the achievement of outcomes

20. Recipient Capacity Associated with Better Outcomes • Addressing issues of recipient capacity was associated with better outcomes About 60 percent of Bank knowledge services reviewed contributed at least in part to developing or strengthening institutions. A large majority of Bank knowledge service and a significant part of IFC Advisory Services contributed to strengthening analytical or policy formulation capacity of recipients.

21. Improving Results of Knowledge Interventions Centerpiece of WBG Current Change Proposals • Main proposals on knowledge and solutions: • Client Engagement: Strengthen client-focused engagement model using science of delivery principles. • Global Practices: Become the best development solutions provider by growing, supporting, and deploying global expertise for agile client delivery. • Invest and fund transformational platforms for knowledge, research, learning, and innovation. • Reinforce knowledge governance, strategy, and behaviors

22. Leadership needed to enhance the Bank global services Governance will benefit from management leadership to encourage knowledge services and develop stronger M&E for knowledge services. An additional challenge will be to bring in the knowledge acquired from knowledge-based partnerships to lower-income countries. Another challenge will require that staff and management have incentives to work on knowledge that are as strong as for other areas.

23. Focus on specific sectors more likely to achieve outcomes than multisector or thematic efforts • Agriculture, education, and financial sector above average • Average in infrastructure • Poverty reduction average, with frequent partial achievement • Private sector/economic policy & governance below average → difficulty coordinating reforms across sectors → if new legislation needed + complicated

24. Financing arrangements not a driver for results, but knowledge in support of lending better results • Reimbursable activities significant in sample • Their results similar to Bank-financed operations • Knowledge services for lending more likely to succeed