Impact Pathways Evaluation: An Approach for Achieving Impact in Complex Adaptive Systems

1. Impact� Pathways Evaluation: An Approach for Achieving Impact in Complex Adaptive Systems Boru Douthwaite Technology Policy Analyst International Center for Tropical Agriculture (CIAT), Colombia Presentation given at WAU on 31st May. 2006

2. Change Models Matter

3. How change happens �Improvements in poverty alleviation, food security and the state of natural resources result from dynamic, interactive, non-linear, and generally uncertain processes of innovation.� EIARD, 2003

4. Impact Pathways Evaluation People plan and implement projects on the basis of their change models - their implicit theories about how the world works If you can improve the theory you can improve the practice, making impact more likely Impact Pathways Evaluation � A participatory approach for: Making practitioners� theories explicit about how they will achieve adoption and impact (impact pathways); Improving them Using those models / frameworks for M&E and impact assessment As a result, contributing to project and program �adaptive management� and thus likelihood of impact

5. History and Current Work Work in Nigeria on Striga Douthwaite, B., T. Kuby, E. van de Fliert and S. Schulz. 2003. Impact Pathway Evaluation: An approach for achieving and attributing impact in complex systems. Agricultural Systems 78 pp243-265 Douthwaite, B., Schulz, S., Olanrewaju, A., Ellis-Jones, J. 2006. Impact pathway evaluation of an integrated Striga hermonthica control project in Northern Nigeria. Agricultural Systems. In press. Current Work (since Oct. 2005) Douthwaite, B., Alvarez, B.S., Cook, S., Davies, R., George, P., and Howell, J. 2006. The Use and Potential of Impact Pathways in the Challenge Program on Water and Food (CPWF). Draft Discussion Paper

6. CPWF Impact Assessment (IA) Project Intervention Adapt and develop methodology Impact pathways; impact narratives; impact pathway evaluation; scenario analysis; extrapolation domain analysis Carry out ex-ante impact assessment Purpose CPWF scientists and management are using IA methods results and methods Goal To contribute to the CPWF fulfilling its impact potential To contribute to the CPWF being perceived as a �coherent, problem-focused research program�

7. Impact Pathways A visual description of the causal chain of events and outcomes that link outputs to the goal (logic model); and Network maps that show the evolving relationships necessary to achieve the goal Implementing organizations; boundary partners; beneficiaries

8. Impact pathways � a more complete picture�.

9. Impact Narrative Text description of the project impact pathways Achieves the integration between the logic and network models Helps with colligation (tracing of logical steps, Roberts, 1996) Helps with the plausibility of ex-ante impact assessment

10. Why develop impact pathways and impact narratives? Carry out ex-ante impact assessment Show the project�s rationale Help communicate what the project is doing More fundable Help with planning Provide a basis for evaluation Starting point for evaluation is a good model of what you think will happen Help to write better project proposals

11. Foundations Impact Pathways; Impact Narratives Adaptation of concepts from Program Evaluation Renger and Titcomb (2002) � problem trees Chen (2005) � program theory Mayne (2004) - performance stories Innovation histories Douthwaite and Ashby, 2005 Social network analysis Cross and Parker, 2004

14. Example of a Problem Tree

15. Turning a problem tree into an objective tree

18. Develop a vision of project success two years after the project ends Work in project groups Take 5 minutes to individually answer the question You wake up 2 years after your project has ended. Your project has been a success and is well on its way to achieving its goal. Describe what this success looks like to a journalist: What is happening differently now? Who is doing what differently? What have been the changes in the lives of the people using the project outputs, and who they interact with? How are project outputs disseminating? What political support is nurturing this spread? How did that happen? Discuss and develop a common vision

19. Example of a Vision

20. Develop a project timeline from when your project started until 2 years after it will end Build a timeline of activities, outputs and outcomes that you from the beginning of the project to achieving the vision It is a story of adoption of project outputs (scaling-out) and the political support that helps it along (scaling-up)

21. Example of a Timeline

22. Impact pathways � a more complete picture �..

24. Advantages of network models Actor-oriented descriptions: observable, understandable, verifiable Captures real-life complexity: We are subject to multiple influences We influence many others And influence works both ways Multi-disciplinary experience with analysis of networks Sociology, political science, psychology, biology, physics, information technologies�

25. A network diagram

26. A plotted network diagram

27. Today�s tasks�.. Identify relevant actors & relationships Develop network diagrams for Your project now Residual network 2 years after project has finished Identify key extension and political support linkages Identify differences between the two networks and discuss implications

29. Selected Feedback from Workshops �I will use Impact Pathways in future design of projects� �The dynamics of the networks is useful to envision the future� �It helps show gaps� �It is good for planning� �It helps explain impact of my project� �Constructing impact pathways should not be one-shot� �The impact pathways should be a living document�

30. Impact Pathways M&E Monitoring and evaluating progress along impact pathways Regularly updating objective tree, timeline and network maps Indicator-based evaluation Most Significant Change to pick up unexpected consequences

31. How change happens

32. Indicators & their limits I am not against the use of indicators. There is a time and place for them Good for tracking activities and outputs, less good at tracking outcomes and impacts MSC is a complementary method, not a replacementI am not against the use of indicators. There is a time and place for them Good for tracking activities and outputs, less good at tracking outcomes and impacts MSC is a complementary method, not a replacement

33. The core of MSC A question: �In your opinion what was the most significant change that took place in �.over the � months� [describe the change and explain why you think it is significant] Re-iteration of the same kind of question �Which of these SC stories do you think is the most significant of all?� [describe the change and explain why you think it is significant]

34. Selecting SC stories In large organisations a hierarchy of selection processes is usually needed Need to decide how many levels Who participates at each level: story providers, their superiors, their peers,.. In large organisations a hierarchy of selection processes is usually needed Need to decide how many levels Who participates at each level: story providers, their superiors, their peers,..

35. Understanding MSC through metaphors Organisations as newspapers, with journalists, sub-editors, editors, senior editors, etc Stories get passed up the hierarchy, but only a few make it to the front page, and only one to the top of the front page Organisations as amoeba, sensing positive and negative experiences and slowly moving to and away from those respectively.

36. Organisations as amoeba

37. Research Question for the CPWF CGIAR Science Council is concerned about high transaction costs. The CPWF should �develop partnerships only in so far as they can be deployed in the short-to-medium run to generate tangible, high impact scientific advances.� (Science Council, 2004, p.1) Innovation and impact emerge from networks not pipelines What types of network should the CPWF attempt to foster to achieve impact? Food security, poverty alleviation, improved health, environmental security More generally, to what extent do network structures affect innovative performance?

38. Summary Change Models matter: People plan and implement projects on the basis of their change models - their implicit theories about how the world works If you can improve the theory you can improve the practice, making impact more likely Impact Pathways Evaluation � A participatory approach for: Making practitioners� theories explicit about how they will achieve adoption and impact (impact pathways); Improving them Using these models / frameworks for M&E and impact assessment As a result, contributing to project and program �adaptive management� and thus likelihood of impact A work in progress, including research on network structures