1 / 14

Objectives

Seminar on selected evaluation methodology issues 评估方法论研讨会 Independent Office of Evaluation of IFAD Beijing, 16 July 2014 国际农业发展基金独立评估办公室 2014年7月16日，北京. Objectives. Discuss selected methodology issues for evaluation of agriculture development interventions

eliot
Download Presentation

Objectives

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Seminar on selected evaluation methodology issues评估方法论研讨会Independent Office of Evaluation of IFADBeijing, 16 July 2014国际农业发展基金独立评估办公室2014年7月16日，北京

  2. Objectives Discuss selected methodology issues for evaluation of agriculture development interventions Collect feedback for the development of the new IFAD/IOE evaluation manual

  3. Context IOE and Ministry of Finance joint Statement of Intend for Evaluation Capacity Development Development of new evaluation manual, to be issued in 2015 Joint PPA of the Environment Conservation and Poverty-Reduction Programme in Ningxia and Shanxi

  4. Key IOE products Corporate-level Evaluations Country Programme Evaluations Project Evaluations Evaluation Synthesis Reports Impact Evaluations

  5. Evaluation criteria used Use of six point rating scale to assess performance

  6. Methodology Issues (1) Data Quality Challenges: No baseline survey; weak M&E system; data mostly on outputs; deficiencies in project log frame; too many indicators Options: Evaluability assessments; reconstruct log frame and underlying theories of changeat evaluation; primary data collection (RRAs, surveys, focus groups, case studies); triangulation

  7. Methodology Issues (2) Assessing efficiency (A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted into results) Challenges: Lack of data on outcomes; difficulty in quantifying benefits; weak data on costs and inputs; inadequate capacity to measure EIRR; not all projects have ex-ante EIRR Options: use of proxy indicators (delays from loan approval to effectiveness; time-over-run, project management costs, cost per beneficiary, final disbursements; etc.); benchmark unit costs (e.g., for infrastructure; CBAs

  8. Methodology Issues (3) Evaluation and hindsight (How to evaluate a project in an evolving context? Is it fair to evaluate a project using today’s standard?) Challenges: New priorities emerge during implementation (e.g., climate change, scaling-up, etc.); evolution of policy and institutional context Options: Learning accountability; retrofitting design during implementation; If project objectives and/or outcome targets were changed during implementation, both the original and revised objectives are taken into account

  9. Methodology Issue (4) Impact Evaluation Definition of impact: The changes that have occurred – as perceived at the time of evaluation – in the lives of the rural people (whether positive or negative, direct or indirect, intended or unintended) as a result of IFAD interventions” IOE Evaluation Manual 2009

  10. Methodology Issue (4) – cont. IOE and impact evaluation IOE has introduced impact evaluations as new independent evaluation products in 2013 1 impact evaluation finalized (Sri Lanka); 1 ongoing (India); 1 planned in the IOE 2015 Work Programme Selectivity framework guiding the identification of the (project) impact evaluation

  11. Methodology Issue (4) – cont. IOE Objectives for Impact Evaluations Assess the results and impact of the programme in a more quantitative manner; Generate recommendations; and Contribute to innovative mixed evaluation methodology development

  12. Methodology Issues (4) – cont. How do impact evaluations provide evidence of the changes caused by an operation? Impact evaluations compare the situation of the beneficiaries who participated in the program (treatment group) against those who did not participate (comparison group), as well as “before and after” the program Mix-methods approach applying quantitative quasi-experimental and qualitative participatory methods: Pre-Post Simple Difference Difference-in-difference Matching Multivariate (Multiple) Regression

  13. Methodology Issues (4) – cont. Impact evaluations: key challenges Complexity of IFAD’s programmes and scarce availability of baseline data Incubation timefor impact measurement Establishment of a counterfactualthat will effectively allow the attribution of impact to the program

  14. Methodology Issues (4) – cont. Impact evaluations: opportunities To improve accountability through assessing not only “if”, but also “how” and “why” the program has, or has not, had an impact and its magnitude To improve systematic and dynamic learning through: enhancing IFAD’s ongoing programs and ultimately better allocate funds across programs feeding results into relevant CPEs and CLEs undertaken by IOE strengthening the internal debate on impact evaluations

More Related