1 / 15

Impact Evaluations and Development Draft NONIE Guidance on Impact Evaluation

Impact Evaluations and Development Draft NONIE Guidance on Impact Evaluation Cairo Conference: Perspectives on Impact Evaluation Tuesday, March 31, 2009 Frans Leeuw Maastricht University & WODC Jos Vaessen Maastricht University & University of Antwerp. Outline. Introduction

fayola
Download Presentation

Impact Evaluations and Development Draft NONIE Guidance on Impact Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Impact Evaluations and Development • Draft NONIE Guidance on Impact Evaluation • Cairo Conference: Perspectives on Impact Evaluation • Tuesday, March 31, 2009 • Frans Leeuw Maastricht University & WODC • Jos Vaessen Maastricht University & University of Antwerp

  2. Outline • Introduction • Methodological and conceptual issues for impact evaluation • Managing impact evaluations

  3. Introduction • Drafting the NONIE Guidance • NONIE uses the OECD-DAC definition of impacts: • Three basic premises: • No single method is best for addressing the variety of questions and aspects that might be part of IE • However, ‘logic of comparative advantages’ • Particular methods or perspectives complement each other in providing a more complete ‘picture’ of impact “The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. These effects can be economic, sociocultural, institutional, environmental, technological or of other types” (OECD-DAC, 2002: 24).

  4. Six methodological and conceptual issues • Identify the type and scope of the intervention • Agree on the objectives of the intervention that are valued • Articulate the theories linking interventions to results • Address the attribution problem • Build on existing knowledge relevant to the impact of interventions • Use a mixed methods approach: the logic of comparative advantages

  5. 1. Identify the type and scope of the intervention • Impact of what vs. impact on what • Impact of what: • Continuum of interventions • ‘Holistic’ vs. deconstruction • Impact on what: • Complexity of processes of change • Levels of impact: institutional vs. beneficiary level

  6. 1. Identify the type and scope of the intervention

  7. 2. Agree on the objectives of the intervention that are valued • What to evaluate should be a balance between what stakeholders find important and the empirical reality of (processes of) change • Intended vs. unintended effects • Short-term vs. long-term effects • Sustainability of effects • Translate objectives into measurable indicators, but at the same time do not lose track of aspects that are difficult to measure

  8. 3. Articulate the theories linking interventions to results • Interventions are theories: opening up the ‘black box’ • Theories are partly ‘hidden’ and require reconstruction • Theory-based IE  continuum of options ranging from: telling the causal story to benchmark for formal testing of causal assumptions

  9. value target variable a c b time ‘before’ ‘after’ 4. Address the attribution problem • Attribution problem: to what extent can results of interest be attributed to an intervention? • Importance of counterfactual analysis

  10. 4. Address the attribution problem • Experimental, quasi-experimental and regression-based techniques have a comparative advantage in addressing the issue of attribution: • Counterfactual analysis • Systematic treatment of threats to validity of claims is possible (and should be done!) • Limitations in applicability

  11. 5. Build on existing knowledge relevant to the impact of interventions • Most interventions are not ‘new’  rely on similar mechanisms of change • Example of types of mechanisms: • Situational mechanisms • Action-formation mechanisms • Transformational mechanisms • Systematic review and synthesis approaches are useful tools for learning about the existing evidence on interventions

  12. 6. Use a mixed methods approach: the logic of comparative advantages • Particular methods have comparative advantages in addressing specific aspects of impact Conceptual framework by Campbell, Cook, Shadish: • internal validity: Is there a causal relationship between intervention and effects? • external validity: Can we generalize findings to other settings? • construct validity: Do the variables that we are measuring adequately represent the phenomena we are interested in?

  13. 6. Use a mixed methods approach: the logic of comparative advantages • Example of how the logic works: impact of incentives on LU change and farmer livelihoods • E.g. randomized experiment can test effectiveness of different incentives on LU change and/or socio-economic effects of these changes (internal validity) • E.g. survey data and case studies can tell us how incentives have different effects on particular types of farm households (strengthens internal validity and increases external validity of findings) • E.g. semi-structured interviews and focus group conversations can tell us more about the nature of effects in terms of production, consumption, poverty, etc. (construct validity)

  14. Managing impact evaluations • Determine if an IE is feasible and worth the cost • Start early – getting the data • Front-end planning is important

  15. THANK YOU jos.vaessen@metajur.unimaas.nl jos.vaessen@ua.ac.be

More Related