Evaluation Methods Training and Capacity Building Programs

1. Evaluation MethodsTraining and Capacity Building Programs Nidhi Khattri Independent Evaluation Group November 17, 2008

2. IEG’s Mandate • The World Bank’s independent evaluation function established about 35 years ago • The goals: • learning from experience • accountability for the achievement of objectives

3. IEG’s Independence: Direct Report to the Board • Direct reporting to the Board of Executive Directors • Headed by a Director-General, Evaluation (DGE) • Appointed by the Board • No World Bank Group position after current position • Evaluations to the Committee on Development Effectiveness (CODE) • Evaluation content not negotiated with CODE/Board

4. IEG’s Links to Bank Management • Bank management has opportunity to comment • Draft Bank Management Response accompanies evaluation • IEG responds to Bank management comments at Board meetings • IEG’ Management Action Record (MAR) reports on management’s progress on actions noted in management response

5. IEG’s Evaluation Products • Project Evaluations • Project assessments (ICR Reviews, PPARs) • Impact studies (e.g., Bangladesh Health, Ghana Education) • Sector and Thematic Evaluations • Often linked to policy revision (e.g., forestry – altered sector policy) • Country Evaluations • Country Assistance Evaluations • Country Impact Reviews (IEG-IFC) • Reviews of CAS Completion Reports • Global and Regional Program Evaluations • Corporate Evaluations • Annual Review of Development Effectiveness (which now includes the Annual Report on Operations Evaluation)

6. Evaluation Approaches • Based on evaluation products • Primarily Objectives Based for Projects and Programs • Outcome • Risk to Development Outcome • Bank Performance • Borrower Performance • Monitoring and Evaluation (M&E) Quality

7. Outcome • The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently • Outcome = Relevance + Efficacy + Efficiency

8. Bank Performance • The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision • Bank Performance = Quality at Entry + Quality of Supervision

9. Monitoring and Evaluation (M&E) Quality • M&E design—the extent to which the project was designed to collect appropriate (input, output, outcome, and impact) data given project objectives and given already available data • M&E implementation—the extent to which appropriate data was actually collected using appropriate collection methods (to ensure data quality) • M&E utilization—the extent to which appropriate data was used to inform decision-making and resource allocation

10. Corporate Evaluation: Annual Review of Development Effectiveness (ARDE) • Annual meta-evaluations that provide a comprehensive assessment of the Bank’s development effectiveness • Draw on IEG’s recent project, sector, thematic, country, and global evaluations • Synthesize lessons that can be used to increase the development effectiveness of World Bank assistance • Highlight the findings of recent IEG evaluations around a common theme

11. Recent IEG Evaluations around Training and Capacity Building • Using Training to Build Capacity (2007) • Avg amount of client training estimated at $720 mio per year (90% through projects, rest through WBI) • Key component in 60% of investment projects, particularly in social, rural, public sectors • 37 (incl. 8 WBI) training programs (Bangladesh, Burkina Faso, Mexico, Tunisia), 6 country surveys of 550 trainees, comparison with other DTIs, etc. • Capacity Building in Africa (2005) • 25% of Bank lending to Africa; $9 bill between 1995-2004 • Public Sector Reform (2008) • 1/6 of Bank lending and advisory support – and increasing

12. Methodology in Evaluation of Training • Key Questions: • To what extent did Bank-financed training have an impact on the capacity of target organizations? • What factors contribute to successful training? • To what extent are such factors present in Bank-financed training? • Main Methods – using objectives-based methodology • Survey of training participants • Survey of training institutions • In-depth field reviews

13. Training Results in Capacity Building Only When Certain Conditions are Met Input Training OutputLearning OutcomeWorkplace behavior change ImpactEnhanced institutional or organizational capacity

14. Effectiveness of Training: What is the Evidence? • Most training resulted in demonstrable learning • But: Individual learning gains poor predictor of impact • Project-based trained lacked basic results measures Outputs Performance Outcomes • About half of trainees surveyed reported substantial • positive changes in work performance Impact on Capacity • 10 project-based programs had significant impact (e.g., • Procurement reform, Community Groups, Exporters, SME) • Best combined project funds + outside expertise • WBI programs not rated due to lack of data Other Findings • Type of training provider not correlated with success • Good training outcomes in both higher- and low- • capacity environments

15. Training Design: What Works? • Adequate diagnosis of capacity gaps associated with • strong client commitment / involvement • Training needs assessment the norm in highly-rated • programs, but often subject to funding constraints (WBI) • Good participant selection requires engagement and • supervision (e.g., IMF Institute, MASHAV) • Poor targeting most important cause for lack of impact • Generally high marks for design & teaching standards • But course length and topic coverage needs to be better matched with capacity building goals (ITCILO,JICA) • Allow time for practical learning techniques / action plans (InWent, JICA, MASHAV) • Provide systematic follow-on support (Motorola, InWent) Targeting of Training Content Pedagogy Workplace Transfer

16. When Training Works: What Matters Most? • Support by managers and peers is key driver for • successful workplace implementation (~90% feedback) • But: 1/3 of trainees didn’t have adequate resources, • incentives or org. support to apply what they learned • Focus needs assessment on organize. bottlenecks and • whether training is indeed the right tool. • Lack of institutional incentives recurring problem in civil • service and highly decentralized training programs; • Stronger incentives at work in training programs for • community and farmer groups, private sector firms • Hi-level support for training key for workplace transfer • Stand-alone training with limited ability for diagnosis, • dialogue, influence and follow-up Workplace Context Incentives Client Commitment

17. Key Internal Inhibitors for Effective Training? • At design, most training programs fail to specify training • objectives and expected performance outcomes, • reflecting lack of broader capacity needs assessments • At completion, performance evaluation rarely done • No feedback / accountability loop >> no improvements • Lack of established standards for training design and • implementation undercuts quality assurance; • Team leaders for project-based training lack adequate • in-house support and voiced demand for more • WBI earns high marks for Country Team consultations, • but collaboration at task level remains rare. Risk of • diffusion of program. Results Orientation Standards Access to Expertise Alignment

18. Persistent Capacity Gaps in Africa Despite Substantial Inputs (IEG, 2005) • 40% of sampled lending operations achieved CB goals, with • better outcomes in roads than in health / education. • CB lacked clear results framework (only 1/3 of projects clear about relationships among individual, organizational. and institutional aspects of capacity) • Weak diagnostics of pol. econ and available country capacity • High fragmentation of efforts; supply driven TA • Training not embedded in broader HR strategies; • Strengthen K-base, operational framework, M&E • Develop sector-specific guidance • Promote country-led approach • Re-assess role / modalities of training • Capacity Building has moved to center stage in AFR CASs • Focus shifting beyond individual skills to institutional support • leadership, donor harmonization and better coherence Key Finding Obstacles Recommended Actions Recent Changes

19. Mixed Outcomes on Public Sector Reform (IEG, 2008) • Public Financial Management: good diagnostics, • indicators, joint undertaking with govts (PEFA) • Tax Administration: strong MoF motivation, good TA • Transparency: widened access to information • Civil Service and Administrative Reforms: lacks good • models, indicators, buttoo important to ignore • Government-wide anti-corruption: key are political • commitment, strong judiciary • Recognize complex political + sequencing issues; • focus on basics first • Prioritize anti-corruption effort on most harmful aspects • Underpin civil service reforms through better diagnosis Successful Themes Reform Challenges Recommended Actions

20. Opportunities Going Forward? • Review & adopt good practice standards for training • programs based on a realistic results framework • Pool evaluation findings to expand practices/lessons • Encourage systematic use in decision-making • Pool expertise around core training and capacity building management practices • Identify case studies of influential training programs • Pilot new forms of collaboration • Bridge the gap between training and other modes of • capacity building • New modalities / increased demand for client-led • training • How training can contribute to development objectives Guidelines Learning & Quality Awareness Scope & Scale Frontier

21. THANK YOU! IEG Website : http://www.worldbank.org/ieg