Mywish Maredia On behalf of

1. Evaluating Impacts of Agricultural Research: Lessons and Challenges Based on the Experience of the CGIAR Mywish Maredia On behalf of The Standing Panel on Impact Assessment (SPIA) of the CGIAR Science Council

2. Outline • Background on CGIAR • Concepts and definitions—ex post impact assessment (epIA) vs. impact evaluation • Overview of the practice of epIA in CGIAR—experience of SPIA • Challenges in assessing impacts of agricultural research and lessons learned

3. About CGIAR • CGIAR=Consultative Group on International Agricultural Research • The CGIAR is a strategic alliance of members, partners and international agricultural centers that mobilizes science to benefit the poor • 64 members: International organizations, governments, and private foundations • 15 centers (also called CG Centers) • FAO, IFAD, UNDP and the World Bank serve as co-sponsors • Annual funding: ~$ 500 million in 2007-2008

4. Mission of CGIAR • To achieve sustainable food security and reduce poverty in developing countries through scientific research and research-related activities in the fields of agriculture, forestry, fisheries, policy, and environment.

5. What is SPIA? • Independent Impact Assessment and Evaluation Group (IAEG) established in 1996, later renamed the Standing Panel on Impact Assessment (SPIA) when it became part of the CGIAR Science Council • It was established by a Task Force of CGIAR in mid-1990s under pressure from donor constituencies • Motivation: • Strong accountability/resource mobilization rationale • Secondary rationale: feedback for priority setting • Need for independence, objectivity and credibility

6. Concepts and Definitions

7. Research for Development (R4D)Results (Value) Chain Time Output Outcome Impact Input Research Objectives Goals

8. Unit of Impact Analysis Different Types of Assessments and Evaluations on Research for Development (R4D) Results Chain Scale Studies that measure the scale of output adoption/ uptake Ex post Impact Assessment as a function of (effect size * scale) Global System Program Program M&E, Impact pathway analysis, Adoption constraints analysis Pilot / Small Impact evaluation studies that measure the effect size Project Time Output Outcome Impact Input Research Objectives Goals

9. What is impact in the context of epIA? • Impact is defined as: Positive and negative, primary and secondary, directly or indirectly, intended or unintended long-term effects that can be attributed to outputs generated by agricultural R4D activity

10. Some Salient Features of epIA • EpIA can only be effectively conducted many years after the completion of the research being assessed (when enough time has elapsed for adoption to take place) • Ex post IA primarily emphasizes the accountability and strategic validation functions: • To systematically build a strong body of evidence about how (or whether) agricultural research contributes to specific development objectives related to enhance food security, poverty alleviation and environmental sustainability. • What has been the magnitude and nature of the impact (positive and negative for various producer and consumer groups, etc.).

11. What epIA is not and does not do? • EpIA is not synonymous with research evaluation (which is broader and occurs earlier in the Results Chain) • It is not appropriate for “real time” feedback and mid-course corrections in research strategies and processes. Other initiatives within the CGIAR such as those associated with “Institutional Learning and Change” (ILAC) do these tasks better

12. Impact Assessments in CGIAR • Done both by centers and SPIA • M&E, process evaluation, adoption constraints analysis – Center • Impact evaluation to measure efficacy and effect size – Center • Studies to document adoption – Center and SPIA (system level) • EpIA – Center and SPIA (system level)

13. SPIA Strategy • SPIA Mandate • To provide CGIAR members with quality information on realized impacts • To partner with centers to enhance epIA and catalyze novel methodologies • To encourage strategic feedback from epIA • SPIA Strategy • Commission independent peer-reviewed studies to advance methodologies, synthesize evidence and draw lessons • Summarize and package information on impacts in digestible forms that nourish investors • Broaden scope of IA down the impact pathway to CGIAR goals and emphasize non-economic indicators using non-economists, implying need for this expertise on SPIA

14. Overview of SPIA Activities • Document evidence of overall efficacy of agricultural research as development assistance Example: SPIA commissioned benefit-cost meta-analysis (Raitzer, 2003). This study: • examined whether the entire investment in the CGIAR over time could be justified on the basis of the benefits derived from its proven (and agreed-upon) major successes. • overall benefit-cost ratio of 9.0 for the $7.12 billion invested (evidence of catalysing substantial additional “multiplier effects” for poor producers and consumers). • study was flagged by a number of donors to the System as the type of assessment required in order to justify continued investments by them into the CGIAR.

15. Overview of SPIA Activities (cont’d) 2. Expanding coverage of IA beyond crop germplasm improvement Examples: • Impact of Agricultural Research on Poverty and Livelihoods (IFPRI coordinated study) Synthesis and 7 case studies (Adato and Meinzen-Dick, 2007) • Natural Resource Management Research Impact Assessment Synthesis and 6 case studies (Waibel and Zilberman, 2007) • Policy Oriented Research Impact Assessment Scoping study (2006) and Synthesis and 7 case studies (2008)

16. Overview of SPIA Activities (cont’d) 3. Raising and establishing standards and good practice in epIA Examples: • Documenting donor demands for and uses of evidence of research impact in the CGIAR – study completed in 2005 • Development of a document providing strategic guidance for conducting epIA of agricultural research report (Walker et al. 2008)

17. Overview of SPIA Activities (cont’d) 4. Promote impact assessment culture where quality epIA is regarded essential in research management and planning Example: • Development of an annual performance measurement system to assess ‘impact culture’ at the Centers

18. Overview of SPIA Activities (cont’d) 5. Systematize and better expose the Impact Assessments done in the CGIAR Example: • Development of a CGIAR impact website to access most recent SPIA and Center IA study reports (http://impact.cgiar.org) • Production of short, 2-4 page ‘impact briefs’ specially targeted to donors and key stakeholders of the CGIAR.

19. Lessons Learned • Good epIA depends on the quality and rigor of analysis that measures the effect size (counterfactual, baseline surveys, methodological rigor) • Documentation of adoption and uptake of research outputs – a necessary step (thus, must be done on a regular basis) • As one moves down the impact pathway, epIA becomes more an analysis of “causal contribution” (rather than “causal attribution”)

20. Lessons Learned (Cont’d) • Rigorous epIA is more difficult for some types of research (e.g., policy) and program goals (e.g., environment) (there is no “one size fits all” strategy) • EpIA requires knowledge to understand the underlying impact pathways, quantitative skills to convert the observed estimates of effect size and scale into costs and benefits, and qualitative assessment and understanding of other potential impacts not reflected in the quantitative estimates

21. Challenges Ahead • Clarifying objectives of and priorities for ex post IA • Developing new IA methods in difficult areas • Moving further down the impact pathway beyond economic measures of impact • Making ex post IA more utilisation focused • Enhance coverage and rigour of Centers’ IA efforts

22. How SPIA is planning to address these challenges? • Plans a study on the key issue --- learning vs accountability: tradeoffs or complements? Elicit views of research directors and senior research scientists to assess actual and potential use of impact related information for decision making 2. Expand on the recent policy research and NRM research impact studies by blending quantitative and qualitative approaches in rigorously establishing (a) counterfactuals, and (b) attribution

23. How SPIA is planning to address these challenges? (cont’d) 3. Launch new initiatives to document impacts of some neglected areas of CGIAR investments: Examples: • Germplasm collection, conservation, characterization and evaluation (biodiversity) • Training and capacity building 4. Broadening the scope of IA beyond partial assessments (particularly aggregate economic assessments) by advancing further down the impact pathway toward indicators that reflect more closely CGIAR goals (e.g., poverty, food security, environmental sustainability)

24. How SPIA is planning to address these challenges? (cont’d) 5. Making ex post IA more utilisation focused (providing strategic feedback and operational learning to improve the design and conduct of future research efforts). How? • Conduct more meta-analyses drawing on individual epIA study experiences to answer: • type of research most effective (greatest impact) • validate (or refute) ex ante predictions • how can spillovers be maximized • Closer interactions with evaluation community (NONIE, 3IE, etc.) and ILAC

25. How SPIA is planning to address these challenges? (cont’d) 6. Helping Centers enhance the coverage and rigour (credibility) of their IA efforts. How? • Closer interactions between SPIA and Center focal points for impact assessment via: • biennial meetings • collaborative projects (NRMR impact study, PORIA study, EIA study) • organizing international symposia (IAAE meetings) • Further refinement of strategic guidance document for conducting impact assessment • Assessment of impact culture across the centers (Annual Performance Measurement System exercise)

26. We Welcome Your Feedback! Thanks