ADB ’ s Evaluation Architecture and Country Evaluations Guidelines

1. ADB’s Evaluation Architecture and Country Evaluations Guidelines Mala Hettige Senior Advisor Independent Evaluation, ADB Presentation at the meeting of Sub-Regional Development Institutions Johannesburg, 27 April 2014

2. Established in 1966 – in Manila, Philippines • 67 shareholding member countries • Provides $ 21b in development assistance • About 760 projects • 5 regional department and PS department • Managed by MDG, 6 VPs • 3000 staff in HQ and 29 RMs + 3 rep. offices • Administrative Budget about $600 million Asian Development Bank (AsDB)

3. Abides by OECD’s 4 dimensions of independence • DGE reports to the Board through the Development Effectiveness Committee (DEC) • Reports public upon DG-IE approval, discussed under Chair, DEC • DGE appointed/removed by DEC – 5 year non-renewable term. • Work Program and separate budget approved by Board • DGE has authority over staff recruitment • Staff subject to backward and forward conflict of interest policy Independent Evaluation @ AsDB

4. Annual Reports Country Assistance Program Evaluations Corporate Evaluation Studies Impact Evaluation Study Thematic Evaluation Studies Project/Program Performance Evaluations Technical Assistance Program Evaluations Major Evaluation Products Real- time Evaluation Studies Project Validation Reports Other Evaluation Reports 4

5. Accountability and Learning 5

6. A database of evaluation lessons • Provides users easy access and search capability • Lessons by sector, theme, topic, country etc. • EVIS is the first system of its kind • Accessible through http://evis.adb.org Evaluation Information System (EVIS) 6

7. Tracks ADB Management actions on recommendations • IED Validates actions taken by Management • Reports this validation to the Board • Computerized and available to staff and Board • Departments/offices update progress on actions Management Action Record System (MARS)

8. Country Evaluations and ADB Operations 8

9. History of Country Evaluations in ADB CAPEs (Country Assistance Program Evaluations) • 1998 first CAPE – Peoples Republic of China • 2002 Mongolia 2002 questioned choices made • 2005 Bhutan had recommendations + criteria ratings • 2006 Indonesia added ratings by sector • 2007 India had contribution to themes+ value added • 2007 China had Strategic and Institutional positioning • 2008 Sri Lanka considered the implications of conflict • 2008 Philippines considered political economy factors • 2009 Cambodia top-down + bottom up assessment • 2010 Lao existing guidelines applied • 2014 Tajikistan assessment of objectives- ongoing

10. Evolving Guidelines for CAPEs in AsDB • From 1998 to 2006 no guidelines • First Country Evaluation Guidelines 2006 • Revised Guidelines issued 2010 • CAPE/ validation of a country self evaluation mandatory prior to approval of a CPS • Now doing a few revisions to the guidelines • Also preparing validation guidelines

11. Good Practice Standards (GPS)for Country Evaluations: Adapting it to ADB

12. Background: GPSon Country Evaluations • GPS for CSPE established in 2008 • Based on OECD–DAC evaluation principles • Draws on the 2007 ECG Review of CSPE • A core GPS criteria allows comparability • An optional GPS criteria helps accountability and learning

13. Application of GPS: of Country Evaluation • Name of study differs and number differs • Coordinated with the upcoming country strategy • Substituted with validations or learning products • About 9-12 months, 25-75 pps, $ 250+, • Lending/non lending, about 10 years (2-3 cycles) • Application of principles, approach taken differs • Overall Rating and Evaluation Criteria followed by about half. But try not to emphasized much

14. Key Aspects of CPSE • Objectives: • Accountability – Board on results over a period • Learning - engage with stakeholders on future CPS • Timing : to feed into the next strategy • Coverage: • Long enough to assess results and • Recent to ensure findings are relevant

15. GPS forCountry Strategy and Program Evaluation • GPS for CSPE established in 2008 • Based on OECD–DAC evaluation principles • Draws on the 2007 ECG Review of CSPE • A core GPS helps allows comparability • An optional GPS helps accountability and learning

16. Evaluation Criteria CoreCriteria Optional Criteria Positioning Coherence Institutional Development MDB Performance Borrower Performance Partner Coordination • Relevance • Effectiveness • Efficiency • Sustainability • Impact

17. Relevance • GPS : Relevance refers to the degree to which MDB strategy and program were consistent to critical needs and long-term plans of the country

18. Relevance –> Strategic Relevance • Can include optional criteria on positioning– how well did MDB respond or anticipate • Evolving development challenges • Priorities of the government • Build on MDB’s comparative advantage • Consider support from other development partners • Also can include coherence – extent to which there were measures to • Foster internal + external synergies within MDB • Foster external partnerships that promote cost-effective division of labor through complimentarities

19. Efficiency • GPS: Efficiency refers to the extent to which the design and delivery of assistance were most cost effective.

20. Effectiveness • GPS: Effectiveness refers to the extent to which the assistance instruments achieved the intentions and objectives set.

21. Sustainability • GPS: Sustainability refers to the likelihood that actual and anticipated results will be resilient to risks beyond the program period.

22. Impact • GPS: Impact refers to an MDB’s contribution to long-term changes in development conditions.

23. Impact => Contribution to Development • Optional Criteria • Institutional Development • MDB Performance • Client Performance • Partnership Performance • Other concerns • By sector? • What about cross sector impacts? • By objectives? • How to attribute and measure? • How to account for unintended impacts?

24. IED Website www.adb.org/evaluation

25. evaluation@adb.org www.adb.org/evaluation https://www.facebook.com/adbevaluation Follow Us https://twitter.com/adbevaluation http://www.youtube.com/evaluationatadb Thank You! http://www.scribd.com/adbevaluation/ http://www.soundcloud.com/adbevaluation/