European Commission Joint Evaluation Unit common to EuropeAid, Relex and Development

1. European CommissionJoint Evaluation Unit common to EuropeAid, Relex and Development Methodology for Evaluation of Budget support operations at Country levelConceptual and Methodological Note

2. CURRENT SITUATION EXISTING BUDGET SUPPORT EVALUATIONS 2004 Tanzania 2006 Joint Evaluation of 7 countries from 1994 to 2004 budget support 2007 Ghana CRITICISM Evaluations addressing mainly THE PROCESS And not enough the CHAIN OF RESULTS • IMPACTS AT THE GLOBAL LEVEL • IMPACTS ON SECTOR POLICIES

3. Expected Outputs of the Budget Support Evaluation • An inventory of the inputs provided through budget support during a given period. • An assessment of the performance of these inputs in terms of direct outputs and induced outputs. • An identification of the changes which have occurred during the period in the partner country and the key causal factors driving those changes related to the budget support. • An assessment of the extent to which budget support may have contributed to the results identified at the outcome and impact levels. • An assessment of the overall relevance of budget support arrangements in the light of the evolving country and sector specific context, the aid policy and the related goals.

4. Comprehensive Evaluation Methodology A Three components approach methodology combining • An evaluation of the budget support itself • An assessment of the budget support related outcomes and impacts of the government strategy • An exploration of the linkages between the budget support process and some outcomes and impacts of the government strategy Focused on the main topics of the budget support • Each topic is associated with one or several Evaluation Questions, Judgement Criteria and Indicators • To ensure harmonisation of the evaluations in various contexts

5. GBS/SBS COMPREHENSIVE EVALUATION FRAMEWORK

6. FIRST COMPONENT EVALUATION OF THE BUDGET SUPPORT Overview of the national context and its evolution Entry conditions for budget support Volume of funds, policy dialogue and conditionalities, technical assistance and capacity building, harmonisation and alignment Assessment of the inputs provided by the budget support Assessment of the direct outputs of the budget support Effects of the identified inputs including effects on transaction costs Assessment of the induced outputs and the corresponding government actions Changes that budget support has induced in comparison to what was expected

7. SECOND COMPONENT ASSESSMENT OF IMPACTS AND OUTCOMES Government • Economic growth • Incidence of income and non income poverty • Degree of empowerment and social inclusion • Evolution of MDG indicators • Achieving sector development objectives for SBS Review of the impacts of the government strategy: specifically those related to the budget support deliverables e.g.

8. SECOND COMPONENT Causal factors • Changes in business confidence and competitiveness • Changes in the use and appreciation by the beneficiaries of goods and services provided by the public sector • Changes in the degree of confidence of the general public sector Identification of the factors that have contributed to the key changes e.g. Identification of the factors external to the government strategy which could have had an impact on the mentioned changes • Regional and international economic • Political environment • Other unforeseen event e.g.

9. SECOND COMPONENT Relationship • Positive linkages between government actions and improved outcomes and • Failure or inappropriate actions which can have jeopardised the quality of the outcome Identification of the outputs of the government strategy which appear to have been of greatest importance in generating change at outcomes level

10. THIRD COMPONENT ASSESSMENT OF THE INFLUENCE OF BUDGET SUPPORT « INTEGRATION AND CONSISTENCIES BETWEEN COMPONENTS 1 & 2 » Comparison of the results of component 1 and component 2 • Improved understanding of the influence of budget support on the institutional and policy changes • Increased consideration of the role of dominant factors in the determination of outcomes and impacts If consistencies: indicate causal links showing positive conclusion for budget support If no consistency: find a possible explanation

11. GBS/SBS COMPREHENSIVE EVALUATION FRAMEWORK Component 1 Component 3 Component 2

12. We must take into account: THE STAKEHOLDERS THE LEAD OF THE EVALUATION THE CONSISTENCY, HARMONISATION AND DISSEMINATION ISSUES THE IMPLEMENTATION THE CAPACITY BUILDING ISSUE HOW TO MANAGE ANDIMPLEMENT THIS EVALUATION?CHALLENGES AND PROPOSALS

13. To whom this evaluation is for? Are the donors the only stakeholders? Proposal 1 All donors, the government and other main stakeholders involved in the budget support evaluated should be part of the evaluation. The evaluation of budget support operation in a designed country should be part of a multi-donor evaluation programme. STAKEHOLDERS

14. Who will take the lead for managing the evaluation? From where will the evaluation be managed? Proposal 2 One of the main donors in the country should be chosen by the other donors to take the lead and consequently for contracting with the evaluators. Special arrangements should be set up to involve representatives in the country. EVALUATION LEAD

15. Proposal 3 To ensure consistency, the headquarter Evaluation Service of the lead donor should be responsible for the evaluation methodology and quality assurance. Each donor can have dedicated persons in his Evaluation Service to go into the countries where the evaluation takes place, at crucial moments. Proposal 4 A network of donors should be set up to ensure harmonisation and dissemination. CONSISTENCY, HARMONISATION AND DISSEMINATION ISSUES

16. Who will do the evaluation? Who has currently the competencies to evaluate budget support and how to manage it? Proposal 5 A team of external independent consultants familiar with the main topics of the evaluation should be contracted for each evaluation. IMPLEMENTATION

17. Do we need a specific training on budget support evaluation? Proposal 6 Make sure that few consultants are immediately operational and have been already trained in existing trainings: Proposal 7 Train the identified people within the Evaluation services of the Member States, some external independent consultants and the main governmental stakeholders of the recipient countries. CAPACITY BUILDING

18. Where are we ? • Issue paper (May 2008) • Steering group at OCDE / DAC level (June 2008) • 1st meeting of a core group (September 2008) • DAC Evaluation Network (18 – 19 November 2008) • 2nd meeting of core group (December 2008) • Common methodological document and standard terms of reference (early 2009) • Testing methodological approach with partners’ agreement (2009)