Outcome Harvesting

1. Outcome Harvesting Ricardo Wilson-Grau 9 February 2012 Beirut, Lebanon

2. The next two hours • Purpose: Introduce you to the principles of Outcome Harvesting as a monitoring and evaluation tool for Outcome Mapping. • Intended results: You will understand the essential principles of Outcome Harvesting and be able to decide when and if it could be useful in your Outcome Mapping work.

3. Outcome Harvesting • A tool for monitoring and evaluating the results of development interventions • Developed since 2003 by me and my colleagues Claudia Fontes, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, and Wolfgang Richert, and currently Barbara Klugman and Natalia Ortiz.

4. 1. International social change networks GLOBAL PARTNERSHIP FOR THE PREVENTION OF ARMED CONFLICT

5. 2. Funding agencies

6. Outcome Harvesting • 0. Focus on utilisation • Design the harvest • Review documentation and draft outcomes • Engage with informants • Substantiate • Analyse, interpret • Support use of findings

7. Outcome Harvesting to meet needs Monitoring Evaluation “I can honestly say that not a day goes by when we don’t use those evaluations in one way or another.”

8. Outcome Harvesting • 0. Focus on utilisation • Design the harvest • Review documentation and draft outcomes • Engage with informants • Substantiate • Analyse, interpret • Support use of findings

9. 0. Focus on usefulness • Primary intended users • Principal intended uses • Useful M&E questions

10. Examples • Management team requires information about programme effectiveness in order to make funding decisions for the next three years.

11. Examples • Management team requires information about programme effectiveness in order to make funding decisions for the next three years. • To what extent did the outcomes we influenced in 2009-2011 represent patterns of progress towards our strategic objectives? • Was our investment in the activities and outputs that contributed to our 2009-2011 outcomes cost-effective? • What? So what?

12. Focus on usefulness compared to OM INTENTIONAL DESIGN INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices STEP 12: Evaluation Plan EVALUATION PLANNING OUTCOME & PERFORMANCE MONITORING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal

13. My comparison INTENTIONAL DESIGN INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

14. Outcome Harvesting • 0. Focus on utilisation • Design the harvest

15. The M&E balancing act

16. The OM concept of “outcome” as changes in social actors

17. Useful Evaluation Question To what extent did the outcomes we influenced in 2009-2011 represent patterns of progress towards our strategic objectives? • Data required to answer • In the light of monitoring and evaluation questions, what data is required and how and from whom will it be obtained?

18. To be harvested CHANGE Behaviour Individual Relationships SOCIAL ACTOR Group or community Policies and practices Organisation Institution Actions, activities

19. Outcome defined • An observable and significant change in a social actor’s behaviour, relationships, activities, actions, policies or practice that has been achieved. .. • … and that has been influenced by the change agent.

20. Outcome example • On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals. • The Government Accountability Project got the incriminating information to the media, leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign. • Source: Evaluation of the Ford Foundation’s Global Governance Programme (2009)

21. Design of Outcome Harvest compared to OM INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

22. My comparison INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

23. Outcome Harvesting • 0. Focus on utilisation • Design the harvest • Review documentation and draft outcomes When? Who? What? Where?

24. Exercise • Please take five minutes to read this short, one-page case study from AWID, the Association for Women’s Rights in Development. Then, with the person seated next to you, identify one outcome: • Who changed what, when and where?

25. MGD3 Fund • Description: During 2011, the Dutch government decided to allocate an additional €12 million to the MGD3 Fund and to launch a new phase of support for the MGD3 with €70 million for the Fund for Leadership Opportunities for Women (FLOW) to strengthen the rights and opportunities for women and girls. • The formulation of an outcome is as specific, verifiable and as detailed as makes sense for the primary intended users and their principal uses.

26. Exercise • Now please identify how AWID contributed to it. • Who changed what, when and where? • What activities and outputs plausibly contributed to the change in the social actor, however partially, indirectly and even unintentionally?

27. MGD3 Fund • Contribution of AWID: AWID compiled the document "Mid-term Summary of the MDG3 Fund Value Added and Outcomes", which was used by the Dutch Ministry in its internal advocacy for the replenishment of the MDG3 Fund. This material along with AWID’s coordination efforts among MDG3 grantees also supported the Dutch gender lobby platform with a strong advocacy tool that they used to lobby the Dutch government to commit these additional resources for women’s rights organizations around the world.

28. Other information? • Significance of the outcome • Collaboration with other social actors • Contribution of other actors and factors • History • Context • Evidence of impact on people’s lives • And so forth – it all depends on the information required to answer the M&E questions.

29. For example • Outcome’s significance: In the light of the impact to which you wish to contribute, why is this outcome important?

30. Outcome example • On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals. • The Government Accountability Project got the incriminating information to the media, leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign. • The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself and it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal.

31. Reviewing documentation and drafting outcomes compared to OM INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

32. My comparison INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

33. My comparison INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

34. Outcome Harvesting • 0. Focus on utilisation • Design the harvest • Review documentation and draft outcomes • Engage with informants

35. 3. Engage with informants • Email • Skype or telephone • In person

36. MGD3 Fund Comment [RW-G1]: Can you be more specific? Who in government? Which agency? • Description: During 2011, the Dutch government decided to allocate an additional €12 million to the MGD3 Fund and to launch a new phase of support for the MGD3 with €70 million for the Fund for Leadership Opportunities for Women (FLOW) to strengthen the rights and opportunities for women and girls.

37. Rita Fund Comment [RW-G1]: Who created this fund? When was it created? Specifically where was it created? • Description: The Rita Fund is created in the United States. The Rita Fund is a woman’s fund which strives to respond to the “funding gap” between donors’ interest and their actual funding  by creating a reliable non restrictive funding source for women’s funds operating worldwide. • Contribution of AWID: AWID’s report Where is the Money for Women’s Rights?, published in 2008, was the source of information and inspiration from which came the idea to create the Rita Fund. Comment [RW-G2]:Is this an appropriate characterisation of “funding gap”? Comment [RW-G3]: Did AWID do something more active to influence the creation of the Fund?

38. Outcome Harvesting to date

43. How much or little detail? • Depends on the users, uses and M&E questions. • Can be one sentence as in the example or a page or more….

44. Outcome example • On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals. • The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself and it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal. • The Government Accountability Project got the incriminating information to the media, leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign.

45. Title: On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals. • Outcome: On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of wide-ranging scandals. Among them: Wolfowitz’s companion, Shaha Riza, received salary raises far in excess of those allowable under Bank rules; Riza received a questionable consulting position with a U.S. defense contractor in 2003 at his direction that resulted in State and Defense Department inquiries; Juan Jose Daboub, Bank Managing Director and Wolfowitz-hire, attempted to remove reference to “family planning” from the Bank’s new health sector strategy; Wolfowitz’s office was responsible for weakening a “climate change” strategy document; Bank Senior Management delayed reporting to Bank staff that a fellow staffer had been seriously wounded in a shooting in Iraq; Bank lending to Africa was deficient and delayed and could only meet lending targets if new loans were brought to the Board for approval without the proper formulation procedures; and that Wolfowitz was trying to broaden the Bank’s portfolio in Iraq over Board opposition, among other disclosures. Within a year, four controversial Wolfowitz appointees had left the Bank, including special advisors Robin Cleveland and Kevin Kellems, General Counsel Ana Palacio and Suzanne Folsom, the director of the Bank’s Department of Institutional Integrity (INT). • Contribution of the grantee: The Government Accountability Project was the recipient of anonymously sent internal World Bank memos and tips relevant to many of the disclosures leading to the Wolfowitz resignation, among them, information that Wolfowitz was negotiating a contract with the resident Iraq Country director, in violation of the Bank’s Articles of Agreement; an email about the shooting of a Bank employee in Iraq and the Bank’s failure to follow its protocol for informing staff about these injuries; payroll records of Shaha Riza; internal documents showing that Daboub, the Wolfowitz-appointed Managing Director, had instructed a team of Bank specialists to delete all references to “family planning” from the proposed Country Assistance Strategy for Madagascar; internal documents indicating that Daboub had ordered the toning down of references to “climate change” in Bank environmental strategy papers; etc. GAP was directly responsible for getting this information to the media – through press releases, leaking documents or corroborating information – leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign. As The Economist wrote on April 12, 2007: “As he prepares to welcome ministers to the bank's spring meetings in Washington, DC, on April 14th and 15th, Mr. Wolfowitz's own toes … may feel a bit toasty. For that, he can thank the Government Accountability Project (GAP), a vibrant participant in America's civil society.” • Significance:The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself, even as its senior officials demand good governance to borrowing countries. In the wake of the scandal, it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal.

46. A story • On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of wide-ranging scandals. As The Economist wrote on April 12, 2007: “As he prepares to welcome ministers to the bank's spring meetings in Washington, DC, on April 14th and 15th, Mr. Wolfowitz's own toes … may feel a bit toasty. For that, he can thank the Government Accountability Project (GAP), a vibrant participant in America's civil society.” • Wolfowitz’s companion, ShahaRiza, received salary raises far in excess of those allowable under Bank rules; Riza received a questionable consulting position with a U.S. defense contractor in 2003 at his direction that resulted in State and Defense Department inquiries; Juan Jose Daboub, Bank Managing Director and Wolfowitz-hire, attempted to remove reference to “family planning” from the Bank’s new health sector strategy; Wolfowitz’s office was responsible for weakening a “climate change” strategy document; Bank Senior Management delayed reporting to Bank staff that a fellow staffer had been seriously wounded in a shooting in Iraq; Bank lending to Africa was deficient and delayed and could only meet lending targets if new loans were brought to the Board for approval without the proper formulation procedures; and that Wolfowitz was trying to broaden the Bank’s portfolio in Iraq over Board opposition, among other disclosures. • The Government Accountability Project was the recipient of anonymously sent internal World Bank memos and tips relevant to many of the disclosures leading to the Wolfowitz resignation, among them, information that Wolfowitz was negotiating a contract with the resident Iraq Country director, in violation of the Bank’s Articles of Agreement; an email about the shooting of a Bank employee in Iraq and the Bank’s failure to follow its protocol for informing staff about these injuries; payroll records of ShahaRiza; internal documents showing that Daboub, the Wolfowitz-appointed Managing Director, had instructed a team of Bank specialists to delete all references to “family planning” from the proposed Country Assistance Strategy for Madagascar; internal documents indicating that Daboub had ordered the toning down of references to “climate change” in Bank environmental strategy papers; etc. GAP was directly responsible for getting this information to the media – through press releases, leaking documents or corroborating information – leading to widespread international coverage, and in turn, inescapable pressure for Mr. Wolfowitz to resign. • The Wolfowitz episode dramatically exposed the weak governance structure of the World Bank itself, even as its senior officials demand good governance to borrowing countries. In the wake of the scandal, it is unlikely that governance at the Bank can revert to the minimal scrutiny and oversight carried out before the scandal. • Within a year, four controversial Wolfowitz appointees had left the Bank, including special advisors Robin Cleveland and Kevin Kellems, General Counsel Ana Palacio and Suzanne Folsom, the director of the Bank’s Department of Institutional Integrity (INT).

47. One liners • In December 2005, the United Nations issues a groundbreakingwhistleblower protection policy. • In August, 2006, GAP, in tandem with the World Bank Staff Association, stopped an attempt by then Bank President Paul Wolfowitz’s office to hurry through a flawed draft whistleblower protection policy. • In January, 2007, the Board of Executive Directors of the African Development Bank (AFDB) approves a whistleblower protection policy. • On May 16, 2007, World Bank President Paul Wolfowitz resigns in a wake of scandals. • The World Bank adopts recommendations for improving the Bank’s investigative unit INT and the Director of INT resigns. • In June 2008, the World Bank passes an improved whistleblower protection policy.

48. Engaging with informants compared to OM INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

49. My comparison INTENTIONAL DESIGN STEP 1: Vision STEP 2: Mission STEP 3: Boundary Partners STEP 4: Outcome Challenges STEP 5: Progress Markers STEP 6: Strategy Maps STEP 7: Organizational Practices INTENTIONAL DESIGN STEP 12: Evaluation Plan EVALUATION PLANNING STEP 8: Monitoring Priorities STEP 9: Outcome Journals STEP 10: Strategy Journal STEP 11: Performance Journal OUTCOME & PERFORMANCE MONITORING

50. Outcome Harvesting • 0. Focus on utilisation • Design the harvest • Review documentation and draft outcomes • Engage with informants • Substantiate