Evaluating the case for AAS using outcome harvesting

1. Evaluating the case for AAS using outcome harvesting Boru Douthwaite, February 2014

2. A key question “Does AAS have the potential to improve the live of millions?” Can we achieve impact at scale? Are we credible? A key question for ourselves A key question for our donors and stakeholders SLG has decided that we should tackle it this year

3. To answer the question • How AAS expects to reach millions is laid out in its overall theory of change • Our evaluation questions become: • What are the types and patterns of outcomes AAS is contributing to? • How do these provide evidence that the ToC is credible, and how do they help us understand why? • Which we will answer through using outcome harvesting

4. What is the AAS ToC? • AAS program-level ToC • AAS will achieve impact through 3 scaling pathways • See Scaling Pathways Chapter of AAS Science Handbook, in particular the description of the three scaling pathways • By applying the AAS Approach

5. AAS’ Program-level ToC Scaling pathways (3) Scaling up and out (of research output) Strengthening capacity to innovate (through research process) Influencing RinD (globally)

6. The manual we will use Download

7. Funding agencies that have used outcome harvesting Source: Wilson-Grau, 2012

8. Outcome harvesting is ‘backwards’ monitoring • Usual practice is to monitor activities, outputs and resulting outcomes • Outcome harvesting starts with outcomes (expected and unexpected) and traces backwards • Outcome harvesting is an evaluation method that uses theory of change, case study methodology and contribution analysis • Which we are going to learn to use in 2014 for potentially wider use in 2015 as a core part of the AAS M&E System

9. Basic definitions • Outcome: a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution. • Outcome Harvest: the identification, formulation, analysis, and interpretation of outcomes to answer useable questions

10. Main players in Outcome Harvesting • Change agent: Individual or organization that influences an outcome (i.e. AAS implementers) • Social actor: Individual, group, community, organization, or institution that changes as a result of a change agent intervention (i.e. actors we seek to influence through our 3 scaling pathways) • Harvest user: The individual(s) who require the findings of an Outcome Harvest to make decisions or take action. This may be one or more people within the change agent organization or third parties such as a donor (i.e. AAS leadership, external reviewers, our donors, communications) • Harvester: Person responsible for managing the Outcome Harvest, often an evaluator (external or internal)

11. Steps in Outcome Harvesting • Design the Outcome Harvest • Identify outcomes and draft outcome descriptions • Engage change agents in further developing outcome descriptions • Substantiate the outcome descriptions • Analyze and interpret • Support use of findings

12. Step 1: Design the Outcome Harvest • Overall design proposed in this presentation; requires agreement and detailing in hubs

13. General design • Users: • Hub teams; AAS leadership; KS&L research theme; • Uses: • Help further develop the AAS approach based on better understanding of what is working and not • To identify early outcomes to help make the case for AAS in the External Evaluation and the proposal for AAS Phase 2; • Useable questions: • What are the types and patterns of outcomes AAS is contributing to? • How do these provide evidence that the ToC is credible (or not), and how do they help us understand why (or not)? • Harvest outcomes in hubs (x5) and globally (x1)

14. Step 2: Identify outcomes and draft outcome descriptions • From reading through existing documentation and story collection (e.g. Most Significant Change in Solomons; January meeting lists) • By asking AAS staff, partners and stakeholders including communities what they think have been significant AAS outcomes For both 1) and 2) use the checklist to ensure a balanced portfolio Who: Hub harvester

15. Format for describing candidate outcomes • Title • Description of: • Outcome (actor(s) who have changed and what that change is) • How the change came about including how AAS contributed. Include a timeline • The significance of the change (future implications) (i.e. Description of the retrospective and prospective ToC for the outcome)

16. Examples of outcomes already identified • Ministry of Fisheries and the Barotse Royal Establishment in Zambia are now better implementing a fishing ban as a result of AAS bringing them together through the stakeholder consultation process and subsequent joint initiatives. As a result the fish size has increased. • The CLCP process in SugotLyte in the Philippines highlighted their abaca disease problem. AAS funded VSU to carry out a rapid appraisal that showed farmers a solution was possible. AAS brokered funding from PCAARD/DOST to bring in new materials while at the same time organized the required community response. Farmers are now …..

17. Step 2a: Screen outcomes • Hub harvester agrees final selection of outcomes with CPL and lead harvester • Screen on the basis of: • Significance of outcome • Size of AAS contribution to the outcome • Realistically looking for a balanced portfolio of outcomes • Question: • How many outcomes are we aiming for? At what level of aggregation?

18. Step 3: Engage change agents in further developing outcome descriptions The guide says: “Harvester engages directly with the change agents to review information extracted from the documentation and to collect additional information on outcomes and the dimensions considered necessary for a complete description” • Elaborate the timeline and causal narrative linking key events together • Interview key informants and reference them in the outcome description • Critically examine the causal claim(s) and seek alternative explanations Use new data to update outcome description

19. Taking the Zambia outcome as an example Change agents: Tabeth, Mendai, key BRE collaborator, key MoF collaborator

20. Taking the Zambia outcome as an example Change agents: Tabeth, Mendai, key BRE collaborator, key MoF collaborator

21. Step 4: Substantiate the outcome descriptions • The guide says: “Confirmation of the substance of an Outcome Description by an informant knowledgeable about the outcome, but independent of the change agent.” • Can be: • A knowledgeable individual recommended by change agent • A panel of experts to review a set of outcomes • Substantiation plan agreed with lead harvester • The hub harvester presents outcome descriptions to substantiator and asks for an opinion • Is the outcome as described and do you agree with AAS’ causal claim (or words to this effect)?

22. Step 5: Analyze and interpret • Analyze the outcome descriptions to answer the evaluation questions • What are the types and patterns of outcomes AAS is contributing to? • How do these provide evidence that the ToC is credible, and how do they help us understand how? • At hub and global level Who analyzes the data?

23. Step 6: Support use of findings • Several routes • Feed findings into AAS M&E for L system in various ways • Publish working paper and article(s) • Opportunity for publishing • Final outcome harvesting report (and subsequent journal article) identified as a AAS science priority for 2014 by SLG • Ensure findings are in the right format for intended users and uses • Link to communications

24. Timing

25. Challenges • Becoming swamped in too many stories • Exercise becomes too big • Coordination and quality • Lack of resources This implies • Keep it simple • Build this into on-going and already planned processes, e.g. community PM&E • This is a pilot – this year we learn how to do this

26. Next steps • Agree to general design (slide 13) • Develop specific hub and global outcome harvesting plans by end of March • That fit with what is already planned • Schedule bilateral conversations between Boru/Rodrigo and CPLs • Set up a outcome harvesting discussion group?

27. Further reading • See AAS Shared to All / AAS M&E / Outcome harvesting

28. Discussion • General design OK? • Users: • Hub teams, AAS leadership; KS&L research theme • Uses: • Help further develop the AAS approach based on better understanding of what is working and not • To identify early outcomes to help make the case for AAS in the External Evaluation and the proposal for AAS Phase 2; • Useable questions: • What are the types and patterns of outcomes AAS is contributing to? • How do these provide evidence that the ToC is credible (or not), and how do they help us understand why (or not)? • Harvest outcomes in hubs (x5) and globally (x1) • How does outcome harvesting link to community PM&E? • How many outcomes do we expect to harvest? What level of aggregation?