Andrew Warner Presentation for European Evaluation Society October 3, 2008

1. Andrew Warner Presentation for European Evaluation Society October 3, 2008 Overcoming intellectual sub-cultures in Development Evaluation � the experience of NONIE

2. The Agenda The international development community have agreed a contract: The contract is more aid and country ownership in exchange for a renewed focus on measuring, disseminating, and acting on results Achieving the results part of this is the key challenge and the great opportunity for evaluators So far, progress has been slow NONIE is an effort to get a coalition of evaluators behind the agenda of more and better impact evaluation

3. The agenda, continued I will speak about the intellectual challenge in doing this The context is that evaluators come from different intellectual subcultures Some disputes are really more about words than anything else Others are rooted in misunderstandings or partial understandings about other methodologies There is a remaining core of real disputes on substantive issues NONIE will release a guidance document by the end of the year that will go some distance in clarifying all of this

4. Confronting intellectual subcultures: two approaches Reduce disputes through a focus on underlying concepts rather than language, and greater understanding of alternative methodologies Insist on an issues-driven agenda

5. Four areas of semantic differences or misunderstandings Use of the term impact Use of the term attribution Contribution analysis, attribution analysis, regression analysis. Redundant, complementary, useful? Randomized controlled trials

6. An issues-driven approach tends to reduce differences Cases where the counterfactual is not the urgent issue Cases where the important issue is variation in treatment; not treatment versus control Cases where selection bias is paramount Cases where evaluability is the paramount issue

7. Attribution � alternative meanings Establishing causality from a group of interventions to the outcome. Measuring impact accurately. Establishing causality of a specific intervention that is part of a package of interventions that, as a group, is known to have an impact. Controlling for other extraneous factors that affect the outcome but are not part of the intervention. Determining which activity is the �new� activity supported when funds are given to a ministry as budget support in a context where money is fungible. Determining the specific contribution of a particular donor in the context where a team of donors all contribute to an intervention.

8. Contribution analysis (Mayne) 1. develop the results chain; 2. assess the existing evidence on results; 3. discuss plausible alternative explanations; 4. assemble the performance story; 5. seek out additional evidence; 6. revise and strengthen the performance story.

9. Randomized controlled trials Lot of baggage associated with this � detracts from the real issues Long-standing statistical/scientific case This is separate from judgments about the value of the information generated