Impact evaluation: international debates and global initiatives

1. Impact evaluation: international debates and global initiatives Howard White International Initiative for Impact Evaluation (3ie)

2. Overview 3ie • Context: the results agenda • The call for evidence • Rigorous impact evaluation • The international response • Reactions

3. The context 3ie • Methodological developments • Results agenda • US: Government Results and Performance Act, GRPA -1993 – USAID adopted six ‘strategic development goals’ e.g. broad-based economic growth, defining outcome indicators for each e.g. per capita growth • UK: Public Service Agreements and Service Delivery Agreements (very similar to Report Cards and Development Indicators in South Africa) • MDGs • Focus on outcomes • Raises questions about attribution

4. GAO’s response 3ie Goals were ‘so broad, and progress affected by many factors other than USAID programmes, that the indicators cannot realistically serve as measures of the agency’s specific efforts’ In 2000 USAID abandoned the goal outcome measures as a basis for monitoring its performance

5. NAO Review of DFID’s performance measurement 3ie • Timescale, exacerbated by data lags • Decentralized programs not aligned with higher-level objectives • Data quality • Impact? Development agencies may not know exactly what impact their efforts are having given the wide range of other agencies and external political, economic and social factors involved.

6. Some general observations 3ie • Use existing data systems, but with a view to data quality and timeliness • Invest in improving existing data systems rather than making new (often parallel) ones • Know the advantages and limitations of different sorts of data (administrative versus survey data) • Avoid indicator proliferation • Don’t be all M and no E

7. Take away message number one 3ie Outcome monitoring is not a valid basis for rigorous performance measurement … hence the need for impact evaluation

8. The initial response 3ie • International agency response • World Bank: DIME and IEG • Inter-American Development Bank* • At country level response strongest in Latin America • Progressa • In Mexico and Colombia social programs legally require impact evaluations to secure funding Note: * See IDS Bulletin March 2008 for discussion

9. ‘When will we ever learn?’:Evaluation Gap Working Group report 3ie After decades in which development agencies have disbursed billions of dollars for social programs, and developing country governments and nongovernmental organizations (NGOs) have spent hundreds of billions more, it is deeply disappointing to recognize that we know relatively little about the net impact of most of these social programs.

10. Evidence of the lack of evidence 3ie • UNICEF review (1995) found 15% had ‘impact assessments’ but ‘many evaluations were unable to properly assess impact because of methodological shortcomings’ • ILO review of 127 studies of 258 community health programs found only 2 which could give robust conclusion on impact on access • Recent (2008) NORAD review found many reports draw impact conclusions on basis of weak evidence (or none at all)

11. Take home message number two 3ie Billions of dollars are being spent with no idea if they are the best intervention to achieve the desired outcome – or even if the help achieve it at all. Rigorous impact evaluation – and only rigorous impact evaluation – will provide the evidence needed for optimal resource allocation

12. The call for rigorous impact evaluation 3ie • When will we ever learn? • An approach which can attribute change in outcomes to the intervention • Requires a well-defined control group • Strongest design is developed at project design stage • Where feasible, a randomized approach should be considered • Poverty action lab • Active promotion of randomized control trials (RCTs) • Evaluations in conjunction with NGOs especially in India and Kenya

13. The international response 3ie Joint AfrEA, NONIE, 3ie impact evaluation conference Cairo May 2009

14. Resources to support IE 3ie • Websites • World Bank (PREM, IEG and NONIE) • Poverty Action Lab • 3ie – www.3ieimpact.org • Training courses • PAL • IPDET • Africa Impact Evaluation Initiative • Opportunities to conduct IEs • DIME • 3ie • Own resources

15. What will 3ie finance? 3ie • Each year two themes and open window • Themes being determined – see 3ie website for details and to submit proposed questions • Southern-driven, issues-based with southern-led evaluation teams • Under each of these three windows • 10-12 ‘quick wins’ • 6-8 large studies • 6 baseline studies • 6 synthetic reviews

16. Some responses 3ie • Only promoting RCTs – not true • RCTs seen as gold standard – not exactly true, issues-led not methods-led, but where feasiblean RCT should certainly be considered (best available method) • A positivist approach – true, but evidence-based policy making is an inherently positivist approach • Another northern initiative – partly true, but 3ie offers great potential for (1) Southern ownership of evaluation agenda, (2) harmonization around quality standards

17. Take home message number three 3ie The current policy framework in South Africa implicitly demands quality impact evaluations, and resources are available to help meet the supply

18. Summary 3ie • The move to performance-based management leads to questions being asked about attribution • Impact evaluation is the approach to be adopted to address attribution • Impact evaluation has to adopt certain technical standards if it is to provide proper evidence • The new international initiatives exist to support efforts to expand programs of impact evaluation

19. Thank you 3ie VISIT www.3ieimpact.org