Challenges to Institutionalizing Impact Evaluation

1. Challenges to Institutionalizing Impact Evaluation Discussion Points* presented at the International Conference on Impact Evaluation Cairo, Egypt March 28- April2, 2009 *By Getahun Tafesse, CIDA –ECCO M&E Advisor

2. Strengths and Challenges to Institutionalizing Impact Evaluation Opportunities • Enormous development challenges in Ethiopia - widespread poverty, chronic food insecurity, HIV/AIDs, illiteracy, environmental degradation etc. • This provides fertile ground for evaluation as intervention resources are limited and different stakeholders are pressurized to demonstrate results • The Government of Ethiopia (GoE) policy statements express commitment to poverty reduction and sustainable development.

3. Strengths and Challenges to Institutionalizing Impact Evaluation Opportunities • Ethiopia has also formally adapted the MDGs as an overarching development guiding framework • The existence of well established sectoral annual review mechanisms • The pool of M&E professionals in the country is gradually increasing as more and more organizations are currently recruiting experts specifically assigned as such

4. Strengths and Challenges to Institutionalizing Impact Evaluation Opportunities • Recent trends show growing recognition and institutionalization of monitoring and evaluation practices: • A growing appreciation of evaluation across different stakeholders and particularly encouragement of participatory evaluation at different levels • Institutionalization of M&E in non-government circles • Increasing conduct of M&E training by organizations • Incorporation of M&E courses in some training programs

5. Actors involved in Evaluation Actors involved in M &E • The Government of Ethiopia (GoE) • National – MOFED, PMO, NBE • Sector ministries • Regional and Woreda administrations • Donors (through projects/ programs) • Bilateral • Multilateral • INGOs/ NGOs (through projects/ programs) • Relief and Development • Advocacy • Professional associations • Academic and Research Institutions • The Private Sector/ business community • Communities

6. Ministry of Finance & Economic Development Central Statistical Authority Welfare Monitoring Unit Sector Ministries Donors INGOs/ NGOs Branch Offices Regional Bureaus Programs Branch Offices Surveys/ Censuses Participatory Poverty Assessments Woreda Desks Facilities Projects Projects Grass Root Communities

7. Welfare Monitoring System in Ethiopia Ministry of Finance and Economic Development Chair National Welfare Monitoring Steering Committee Ministry of Agriculture and Rural Dev’t Member Ministry of Health Member RCBP interest could be reflected here Ministry of Labour and Social Affairs Member Ministry of Women Affairs Member Ministry of Education Member Ethiopian Road Authority Member National Welfare Monitoring Technical Advisory Committee Same as Above INGOs/ NGOs Welfare Monitoring Unit Central Statistical Authority Data Users HHICES Sector Reports Welfare Monitoring Survey PPAs

8. The Status Quo in Impact Evaluation • Impact Evaluation is not a common practice • Some sorts of Evaluation are practiced mostly to fulfil donor requirements Can Impact Evaluation meet the sense of urgency that characterize the need for development intervention in low income countries? • Frequently development needs require urgent/ immediate assistance • Programming in developing countries is geared towards fulfilment of basic needs • Choice of area of Intervention is hardly questionable * Difficult to make the main thrust of development programming as Experimentation

9. The Status Quo in Impact Evaluation Is there national ownership in Impact Evaluation? • Impact Evaluation implies there is a preconceived desired State • Whose definition of ‘ development ‘ matters • If desired state is imposed, ownership to IE is lost

10. The Status Quo in Impact Evaluation Is the Cure for development problems known? • Good governance – participation, accountability, transparency… • Investment on education, health, agriculture, road… • Equity, peace, security… • Ownership, partnership, harmonization… *But the degree of effectiveness of a specific programming in these areas is not easily known

11. The Status Quo in Impact Evaluation The Impact evaluation Dilemma • There is sense of urgency for development programming • Appropriate cure depends on Impact evaluation/ experimentation So, the focus of Impact Evaluation in LDCs should be • Not on identification of Areas of investment • Rather on methods of delivery The Cure is known (the vaccine is identified). The main question is how to deliver best.

12. The Status Quo in Impact Evaluation Methodological Challenges How to make the Case of IE strong? • Demonstrating practical benefits of Impact Evaluation • Resources are scarce – how much to spend on something that is intuitively known is good • Flexible and easy IE techniques/ methods that are less costly in terms of time and resources • IE aimed at guiding/ improving decision making • Show possible alternative use of resources

13. The Status Quo in Impact Evaluation Recommendations • Make Impact Evaluation a parallel endeavour/ not a major thrust of development programming – experimentation on a small scale • Impact evaluation on key programs without disturbing regular programming, i.e., without creating discontinuity in program • Impact Evaluation as a Second Stage Experimentation with the focus being on Approach to delivery, exploring practical alternatives • Impact Evaluation in developing countries should have a strong component of comparison element among known strategies • From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources

14. The Status Quo in Impact Evaluation Recommendations • Impact Evaluation in developing countries should have a strong component of comparison element among known strategies – not simple and full focus on the specific program that is the subject of evaluation • IE should be designed to aide decision making on ‘so what’ • From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources

15. Strengths and Challenges to Institutionalizing Impact Evaluation Strengths • GoE national Reports have greatly been improved in quality (depth of analyses) and coverage (sectors, sub-sectors, themes) due to • Growing demand for and use of such reports • DAG financial and technical support extended to MoFED & use of professionals (consultants) for data analyses and report production • Inclusion of Governance, foreign Aid, MDGs, Environment, Gender, etc. themes in the reports although not to sufficient degree • Implementation of regular household surveys (HHICE, WMS, DHS) • Improved capacity and performance on the part of CSA • Improved capacity and performance on the part of selected sectoral line ministries • Great improvements in sectoral reports, especially Health and Education aided by • Sector Management Information Systems • Annual Review Mechanisms • High political commitment (Health, for example) • Improved standardization, rationalization an harmonization of indicators, data collection and reporting procedures

16. Strengths and Challenges to Institutionalizing Impact Evaluation Progresses • Strong and growing capacity in statistical data collection • CSA’s impressive data collection program • Annual and periodical regular surveys • Long experience and institutional capacity • Civil service reform including business process reengineering (BPR) • Streamlined tasks and responsibilities • Result oriented work planning • Good practice of contracting out data collection and analyses • Supported by growing # and capacity of private consultants

17. Strengths and Challenges to Institutionalizing Impact Evaluation Opportunities • Strong culture of collaboration by beneficiaries in responding to studies and gradual development in their level of active participation and articulate responses

18. Strengths and Challenges to Institutionalizing Impact Evaluation Opportunities • Availability of administrative data by sector ministries (eg. education, health etc), and survey data by Central Statistical Agency • Cooperation among stakeholders (government and non-government alike) to share data available • Structured societal organizations established down to small community level. • Some level of established practice in using evaluation for planning purposes…

19. Strengths and Challenges to Institutionalizing Impact Evaluation Challenges • Lack of informed debate on local development perspectives and relevant evaluation conceptual frameworks and approaches; which leads to: • Lack of consensus on development concept and measurement criteria • Lack of consensus on evaluation concept and criteria • Uniform application of evaluation techniques and lack of adoption to specific cultural and behavioral contexts

20. Limitations/ Challenges • Widespread traditional management practice that gives focus to counting activities and outputs rather than focus to assessing higher level results, i.e., poor result based management practice. This is particularly reflected: • In absence of baseline data • Poor feedback mechanism • Lack of informed decision making or poor linkage between assessment and decision making

21. Strengths and Challenges to Institutionalizing Impact Evaluation Challenges • Data collection, analyses and reporting aimed at demonstrating achievements and less focus given to analyses of constraints and challenges • Significant discrepancies between administrative and survey data

22. Strengths and Challenges to Institutionalizing Impact Evaluation Limitations • Lack of linkage/ integration across different sectoral M&E systems • Different timing • Different level of reporting • duplication • sectors at varying stages of capacity and performance in evaluation practice • poor practice of verification methods/ triangulation of data from different sources

23. Limitations • Generating Compelling and evidence based results attributable to programming • Large number and varying quality of indicators used in GoE reports and challenges to discern overall progress • Missing reference comparisons or lack of standard reference points • Missing indicators w.r.t. gender, disadvantaged groups (disabled, destitute, etc.)

24. Limitations/ Challenges • Routine data collection at lower level cumbersome and unsystematic • Simplifying data collection & ensuring timely use of data • The progressive data aggregation at each higher level not necessarily conducive to data analyses • No systematic integration of national data collection activities • The need to support national strategy for the development of statistics • Integration of data collection systems (within & outside sectors)

25. Limitations/ Challenges • Poor feed back and linkage with planning and decision making • weak data producer and users common forums and linkages

26. Limitations/ Challenges • Lack of standardization of survey methods, definition of indicators and measurement tools. • Poor recognition given to evaluation importance as reflected by: • Poor integration of evaluation approach in programs/ projects design • Lack of earmarked budget for M&E • Lack of earmarked human resource for M&E in established structures • M&E task usually undertaken as an add-on task

27. Limitations • Generation of data disaggregated at woreda level • Different levels of reporting across different sectors • No regular complementary qualitative information (PPA, citizen card, etc.)/ public opinion

28. Limitations/ Challenges • Poor Maximizing of benefits from data analyses • Gender disaggregated data • Thorough/ in depth analyses of data sets • Timely analyses of data • Inadequate practice and capacity to review and enforce good ethical standards in the undertaking of evaluation • Teaching in evaluation not well recognized and integrated in curricula of different disciplines

29. Limitations/ Challenges • Poor institutional capacity especially at lower levels • Fragile and over loaded • No earmarked human and financial resource – M&E is usually an add-on task • High staff turn-over

30. Limitations/ Challenges • Lack of agency/ home, networks for • developing and disseminating knowledge on evaluation approaches, tools, and best practices • adoption and popularization of internationally set goals and commitments (ex., MDGs, Conventions, Declarations ) • adoption and popularization of established methodologies • sharing of experiences and exchanging ideas

31. Food SecurityMonitoring & Evaluation

32. GoE Food Security M&E System • The FSCB has an overall responsibility for programme M&E • Food Security Programs M&E Plan • different stakeholders were involved in the process & many acquired opportunity to comment • Simple and Practical • 4 Principles were applied: • Simplicity –vs- Utility • Process –vs- Product/ outcome • Decentralization –vs- Accountability • Participation –vs- Rigour

33. GoE Food Security M&E System • Food Security Programs M&E Plan • Result Frameworks (recently revised based on two 2-day workshops involving GoE & Donors) • Monitoring Formats • Activity Reports (from community up to federal) • Quarterly financial and procurement reports developed by FSCB in accordance with GoE accounting procedures and PSNP Procurement Guidelines. • Focal persons responsible at different levels Training given to focal persons on Monitoring Formats

34. GoE Food Security M&E System • Food Security Programs M&E Plan • Programme Description • FSP Logical Framework • M&E System: Objectives and Approaches • Institutional Roles and Information Flow • Monitoring Guidelines and Methods • Evaluation Guidelines and Methods • Human Resource Needs and Training Plan • Reporting Formats

35. GoE Food Security M&E System • Monitoring and Evaluation Technical Task Force • Comprises members from GoE and Donors financing food security programs • Meets every two-weeks • Oversees the implementation of the M&E plan • Reviews study designs and mobilizes the necessary technical assistance and capacity building resources

36. GoE Food Security M&E System • JCC - meetings every fortnight to discuss and decide on various issues related to the implementation of the program (resource flows to the beneficiary, targeting issues, capacity building and other pertinent issues as they arise). • Rapid Response Mechanism field monitoring (usually with both FSCB and donor representatives) undertaken on monthly basis to examine ad hoc issues and constraints as they arise. Team reports identifying issues and recommendations are presented to the JCC for consideration.

37. GoE Food Security M&E System • An Information Centre has been set up and staffed in the FSCB to regularly follow up on problems at woreda and regional levels (particularly with respect to the flow of funds to woredas and payments to beneficiaries). • Joint Implementation Support Missions are conducted twice each year (May and October) to review progress with program implementation per se and with related capacity building actions (e.g., financial management, public works, woreda planning and quality control, etc.). The FSCB presents financial and technical progress reports during the missions, as well as reports prepared by the RRM Teams. Additional reports may be produced as required, including reviews of procurement.

38. GoE Food Security M&E System • Surveys and Studies • Baseline Survey • Annual Survey • Public Work Reviews • Food Aid Assessments • Other specific studies • Targeting • Institutional Assessment • Impact

39. GoE Food Security M&E System Other data sources: • Annual Agricultural Statistics • Crop forecast, Actual crop assessment • Woreda level Ethiopian Agricultural Sample Enumeration (EASE) • Agricultural Census • DPPA Early Warning System • Vulnerability Profiles