Experiences in Impact Evaluation: The PEMA Perspective

1. Experiences in Impact Evaluation: The PEMA Perspective

2. Mandate • PEMA (Centre for Project Evaluation & Macroeconomic Analysis) was established in January 2003 as an operating unit of the Ministry of International Cooperation

3. While a basic mandate is impact evaluation, this mandate has been extended during the past three years to: a) Help upgrade the M&E capacities in key ministries b) Coordinate the M&E reports prepared by various U.N. agencies in Egypt to assess progress made annually in achieving the objectives under the United Nations Development Assistance Framework (UNDAF) c) Contribute to OECD efforts to survey progress towards better aid effectiveness

4. PEMA undertakes evaluations of development projects in order to: • Ascertain their impact on target beneficiaries and the sustainability of such impact • Draw lessons learnt from evaluation and put these at the disposal of policy and decision makers to assist them in taking measures to improve the efficiency and effectiveness of assistance received

5. PEMA has completed the evaluation of 31 projects in various sectors Current staff of the Centre compromises twelve researchers and the executive director

6. Steps for undertaking impact evaluation Once a project is selected, the following steps are undertaken: • Desk study of documents • Propose evaluation questions and methodology • Data gathering (fieldwork) • Content analysis • Draft report writing • Roundtable meeting is held to receive stakeholders’ comments • Final report is sent to stakeholders and to Minister of International Cooperation, whereby follow-up actions take place

7. Findings on impact evaluations in various sectors

8. Educational Sector • Girls’ access to primary (basic) education is enhanced by providing comprehensive packages of community services • Projects implemented for improving teaching methods in primary schools are not sustainable due to weak monitoring mechanisms Vocational Education: • Businessmen (private sector) support is crucial for the success and sustainability of vocational education programs, particularly for funding training in specialized areas. • Many of the graduates of vocational schools do not pursue a vocational career as they enter universities to obtain academic degrees

9. Small and micro-enterprise development • Building the capacities of NGOs and CDAs is crucial for creating efficient systems for operating micro-credit loans • High charges of loans and disregard of the grace period for loan repayment impede the sustainability of SMEs • Insufficient follow-up surveys to determine the impact of loans on beneficiaries, and to identify measures that could further enhance sustainability and effectiveness of loans • Insufficient coordination between funding agencies results in the implementation of several SME projects in the same areas, thus reducing demand for loans

10. Health Sector • Embedding project activities into the existing operational structure of health system is essential for ensuring sustainability • Systems for collection and analysis of data on health services are required for effective supervision and delivery of services • Highly centralized system of distribution of medical resources (medical staff and medical supplies) results in ineffective distribution • Low salaries and inadequate financial incentives lead to shortage of medical staff and health workers

11. Agricultural Sector Insufficient public facilities discourages beneficiaries from settling in reclaimed areas Insufficient irrigation water impedes application of modern farming practices in some of the reclaimed areas Limited financial and technical capabilities of farmers associations after project termination

12. Lessons learned from impact evaluations More emphasis should be given to project design: project objectives; outcomes; means of verification and implementation strategy Some projects do not set technical and financial mechanisms required for sustaining project outcomes In some projects, responsible bodies do not establish new units or appoint permanent staff to implement and monitor project activities

13. Capacity building activities should be relevant and based on the needs of trainees Setting criteria for selecting trainees and ensuring application of criteria are important for ensuring effectiveness of capacity building activities Executing bodies should have more flexible systems of work by encouraging trained personnel to apply what they learned from the project

14. It is crucial to establish budget lines within ministries for capacity building activities Bad management of project's time leads to implementation of project activities in the remaining time of the project. Therefore, activities are implemented at "less than acceptable" level. A correct budget estimate for project activities is essential to avoid deficit or loss of funds How do target beneficiaries perceive of the project and its outcomes is an important indicator of project effectiveness