Impact Evaluation Clinics for: The Institutional Support to Rural Sector Public Service Administration (AMOASR) Progra

1. Impact Evaluation Clinics for: The Institutional Support to Rural Sector Public Service Administration (AMOASR) Program in Cameroon Case study 2 NONIE meeting in Lisbon October 1, 2008 Network of Networks on Impact Evaluation

2. Location of Cameroon in Central Africa Network of Networks on Impact Evaluation

3. AdministrativeOrganisationof the country • 10 provinces • 58 Divisions - départements • Subdivisions – sous-préfectures • Districts Network of Networks on Impact Evaluation

4. Section A: Presentation of project Network of Networks on Impact Evaluation

5. Outline of presentation • Description of the problem the program deals with • Objectives of the program & results chain • Program theory/logic model • Program activities & budget allocations • Description of the target groups & selection criteria • A Summary of Administrative Data on the Program (financiers, implementation arrangements, budget, evaluation budget share) • Challenges to Evaluation of the Impacts of AMOASR Program Network of Networks on Impact Evaluation

6. What is the problem? • 1980’s: Huge budget cuts due to economic & public fiscal hardship • 1987 to Date: Attempt to focus on regulatory functions rather than on productive and redistributive functions – need for redeployment • Agriculture contributes 22 to 30% share of GDP and 60-75% of employment, and was therefore targeted to be the main drive of poverty alleviation efforts of the Government under the HIPC initiative • 2002: Organizational assessment of the Ministry of Agriculture & Rural Development (MINADER) & Ministry of Livestock, Fisheries and Animal Industries (MINEPIA) Network of Networks on Impact Evaluation

7. What is the problem? • Policy and program design is not informed enough by evidence and hard data • Weak coordination of policy implementation • Lack of appropriation of national policies and directives • Ineffective programming in terms of changes achieved at farmer level (work plans are not results-oriented) • Few activities and related budget are implemented as planned • Decision makers lack information to adjust implementation accordingly to changes in the context and observed performance Network of Networks on Impact Evaluation

8. Objectives of the program • Development Objective: • Strengthen the institutional capacity of MINADER and MINEPIA to implement the Agriculture, Livestock and Fisheries component of the rural sector development strategy (SDSR) • Specific objectives: • Strengthen the audit, monitoring & evaluation technical capacity of the two ministries; • Support the renovation of the agricultural, pastoral & fisheries statistical system; • Strengthen the organizational capacity of the services at national, provincial and divisional levels in the two ministries; • Promote dialogue among the different groups of players in the rural sector. Network of Networks on Impact Evaluation

9. Results Chain • Impacts: • Better governance at national, provincial and divisional levels in MINADER and MINEPIA administrations • Increased impacts of public development interventions in the rural sector (Farmers and their organizations, private agric. firms and civil society organizations) • Outcomes: • Increased consumption rate of domestic budget and absorption rate of aid funds; • Results-oriented (annual & medium term) plans and budgets implemented at various levels of the public services of the target ministries; • Institutionalized dialogue between public decision makers and farmers organizations • Increased public service output delivery rate. • Outputs: • Training services • Technical assistance (design of procedures and templates; development of tools and software; communication packages) • Budget support Network of Networks on Impact Evaluation

10. Inputs Activities Outputs Outcomes Impacts Staff Technical assistance (statistics, M&E units, planning units, procurement and budget units) Systems, procedures and tools in place, including software Strengthened planning, programming, budgeting and monitoring capacity of MINADER & MINEPIA services C2D Funds Better governance of MINADER & MINEPIA services Matching Funds Procurement of goods and services for MINADER & MINEPIA services Training services (sessions) Results-oriented (annual & medium term) plans and budgets implemented at various levels of the ministries Budget available in time to services Goods available in time to services M&E of the AMOASR Program Increased consumption rate of domestic budget and absorption rate of aid funds Increased farmer production & revenues Overall Management of the AMOASR Program Other contributing outputs (incentives & motivation; norms & values; leadership style; etc.) Increased public service output delivery rate Improved rural standard of living Program Theory/Logic Model Network of Networks on Impact Evaluation

11. Project activities and yearly budget allocation (in USD)1 USD = 453 CFAF or XAF Network of Networks on Impact Evaluation

12. Target groups/beneficiaries • Services involved in the PPBS chain in the • Ministry of Agriculture & Rural Development (MINADER) • Ministry of Livestock, Fisheries & Animal Industries (MINEPIA) • 235 services in the 2 ministries Network of Networks on Impact Evaluation

13. Target groups/beneficiaries • In the Ministry of Agriculture & Rural Development (MINADER): • The general inspectorate at the national level • 05 directorates at the national level • 30 services at provincial level (3 in each of the 10 provinces) • 116 services at division level (2 in each of the 58 divisions) Network of Networks on Impact Evaluation

14. Target groups/beneficiaries • In the Ministry of Livestock, Fisheries & Animal Industries (MINEPIA): • The general inspectorate at the national level • 04 directorates at the national level • 20 services at provincial level (2 in each of the 10 provinces) • 58 services at division level (1 in each of the 58 divisions) Network of Networks on Impact Evaluation

15. Selection of participants • All Services involved in the PPBS chain • Personnel in place (no recruitment) Network of Networks on Impact Evaluation

16. Summary Administrative Data on AMOASR • Financiers: • Government of Cameroon (?%) • France’s Ministry of Foreign Affairs/C2D (?%) • Date of Approval: Feb. 2008 • Start of Operation: August 2008 • Total Budget: USD 15 748 189 • Evaluation Budget: USD 397 350 (2,52%) • Administrative Arrangements: • Steering Committee • A Central Implementation Unit (7 Staff) • In-Service Focal Points (at beneficiaries level) Network of Networks on Impact Evaluation

17. Data Availability • Very few data exist on the operations of the services in MINADER & MINEPIA including the routine administrative data: • Number and profile of personnel • Number and profile of the clients served – farmers, investors, other administrations, etc. • Description of services delivered Network of Networks on Impact Evaluation

18. Section B: Impact evaluation of project Network of Networks on Impact Evaluation

19. Evaluation framework and context • The development outcome that the project is seeking to address seems to be improved farmer production and revenues • Would the government and donor be satisfied if the governance of administration was improved but farmer welfare remained the same? • If not, complementary activities need to be considered to attain this ultimate objective • Who are the target group to be evaluated? • Currently, the project targets the capacity building for back office government personnel, not the «ultimate client«, the farmer. Network of Networks on Impact Evaluation

20. Project outcomes and indicators • Outcomes and indicators for the project as designed • OUTCOME: Back office delivers effective and timely services to front line staff • Through improved capacity of back-office (indicator) measured by front office staff perceptions • Through compliance of plans with result-based standards (indicator, desk review of standards, based on expert assessments of quality of services) Network of Networks on Impact Evaluation

21. Impact Evaluation question • Has the project’s inputs (training, goods and services etc.) improved back office delivery of services to front line staff? • The implied theory behind this evaluation question is that improved back office service delivery -> improved front line services to farmers -> improved farmer productivity and welfare

22. Inputs Activities Outputs Outcomes Impacts Staff Technical assistance (statistics, M&E units, planning units, procurement and budget units) Systems, procedures and tools in place, including software Strengthened planning, programming, budgeting and monitoring capacity of MINADER & MINEPIA services Better governance of MINADER & MINEPIA back office services C2D Funds Matching Funds Procurement of goods and services for MINADER & MINEPIA services Training services (sessions) Results-oriented (annual & medium term) plans and budgets implemented at various levels of the ministries Improved public service delivery by frontline services Budget available in time to services Goods available in time to services M&E of the AMOASR Program Other external factors (weather, price of the inputs, marketing infrastructure) Increased farmer production & revenues Overall Management of the AMOASR Program Other contributing outputs (incentives & motivation; norms & values; leadership style) Improved rural standard of living Program Theory/Logic Model (revised) Network of Networks on Impact Evaluation

23. Impact Evaluation Design • Who will be surveyed and how? • Unit of analysis is the service unit (both central level, and samples from provincial and division level) • Information collected will have to be for the entire MINADER and MINEPIA services including front line services • At provincial and division level, stratified random sample of units • Mixed methods approach (for each selected unit): • Focus group discussions to look at perceived barriers to set hypotheses, and use findings to design questionnaire for quantitative and qualitative surveys • Survey of front line service looking at perceived quality of back-office services • Desk review (expert assessment) of compliance with service standards and implementation • Baseline survey to assess current performance level • Follow-up surveys after 40% coverage of services, at mid-term and at completion • Follow ups use the rollout of service delivery to establish the counterfactual of impact on back-office services when project services have not been delivered Network of Networks on Impact Evaluation

24. Impact Evaluation Challenges • Program Spillovers: • All projects implemented by MINADER/MINEPIA: the more effective is AMOASR, the more are impacts of RDI of those ministries • High mobility of personnel (trained ones may leave to others ministries; retirement) • Sources of Contamination: • Public Service Wide Modernization Project – Limited Influence • No similar initiatives within the target ministries Network of Networks on Impact Evaluation

25. Estimation methods, and dealing with challenges • Quantitative survey has questions both measuring perceptions, but also including demographic and unit-specific characteristics (including geographic location, level of education/background of front-line etc.) • Identify sources of variation among surveyed units (using country knowledge and focus groups) • A robust quantitative questionnaire will be critical to evaluation success • The non-quantitative technical evaluation of services will also yield sources of variation • Difference-In-Difference approach to take care of some selection bias, and to take advantage of rollout feature. • Follow up focus group to tease out the ``why´´ • Contagion issue a problem – do units know that others have received program services and thus have greater expectations about program services? • Include question about how much they know of program Network of Networks on Impact Evaluation