State of Sustainability Planning & Evaluation in Community Health

1. State of Sustainability Planning & Evaluation in Community Health Sustainability Framework (SF) 101 April 21, 2011 IntraHealth with the SHOUT Group

2. IntraHealth Renée Kotz, Sara Margolis-Pacqué, Frances Thornhill, Kris Horvath ICF Macro Jennifer Yourkavitch, Ilona Varallyay, Kiersten Johnson, Leo Ryan Acknowledgements

3. SF 101 - Outline • Part 1: Introduction--“It’s complicated” • Part 2: Practical Recent M&E Experience • Part 3: SF 101: • Planning: Small but Critical Steps • M&E: Do we count all trees to measure a forest? • Conclusion

4. It’s not complicated; it’s complex Part 1 - Introduction

5. Current Context Exhibit A: Global Health Initiative • “The challenge of the next decade and beyond is to take these impressive accomplishments to the next level by helping countriesachieve long-term sustainability in their health services.” • “Building on a long tradition of U.S. government global health leadership and the unprecedented level of commitment manifested in recent years, the Obama Administration’s Global Health Initiative has the opportunity to move global health to a new level of effectiveness, with a vision of long-term sustainability led by partner countries.” GHI— [Implementation of the Global Health Initiative: Consultation Document]

6. Do we need a system perspective?E.g. Urban Health System, Bangladesh (Concern) ADB, DfID, USAID, etc. MOLGRD MOHFW City Government Chairman MOHFW District & Sub-District Health Platform (MESPCC) Health Inspector (in absence of Med Off.) Health Department NGO Health Providers Govt & Private Health Facilities Social & religious leaders Commissioner Private pharmacists Community organizations Ward Health Committee Teachers Youth volunteers Traditional health providers Civil Society

7. Sustainability in HSS: a Complex Equation? “Health -programme sustainability is the ultimate manifestation of a complex web of inter-relations between health concerns, stakeholders, resources, and actions analogous to an ecosystem.” Gruen et al. The Lancet. 2008 Illustration inspired by Samir Rihani: “Complex Systems: Theory and Development Practice. Understanding Non-Linear Realities.”

8. Questions I don’t understand: Is it sustainable? Can you ensure it will be sustained? Who will take over after you leave? Common Evaluation Questions about Sustainability • Questions I understand: • Will you leave partners with a better chance of facing what will come after you leave? • Are you at least improving expectations for continued progress in the context? • Are you thinking “within a system?” [a.k.a. is the “Sustainability Scenario” coherent from an internal development logic?] • Do you have information about where your efforts are heading? • Are your partners mastering the processes which produce the results? • Do the M&E systems inform the local actors about their progress?

9. Evolving Understanding of Sustainability— A Suggestion: • An (emerging?) property within a local system which allows interdependent actors to maintain and improve the health status of the system’s (vulnerable) population through negotiated and coordinated social interactions, allowing the expression of their respective and collective capabilities.

10. Implications of this definition • Other definitions (i.e. financial viability, organizational capacity, policy alignment), identify potential determinants of sustainability • Projects can contribute to the process, or hinder it, but ultimately neither control nor define it. • We can, however, measure progress on hypothesized determinants of sustainability during our projects • More than one configuration can support the same sustainable outcome • Allowing systems to find their own new equilibrium might be more important than enforcing the best approach

11. Understanding the Role of External Agents in Building Sustainability In a Linear Process In a Complex Process Contribution:Ownership? Negotiation? Alignment? Shared Accountability “advance capacity & processes” Role of “Agents” Agents have strategies “models” emerge / constructed “Donors and Implementers as Types of Agents.” Evidence? Challenging ; Growing Interest; More robust? • Attribution:Input->Process->Output->Outcome->Impact • Direct Accountability • “ensure sustainability in X years” • Fits Top-Down Planning • Caveat:“involve stakeholders” • pre-determined “models” • “We come, we fix, we go. Hey! They ‘got it’!” • Evidence? • Partial, sub-systems, very thin.

12. Prospective(USAID/NEPAL) Point in time evaluation of progress toward sustainability by supported districts Prospective & Retrospective(USAID/CSHGP/Concern Bangladesh) Planning for sustainability What has been sustained? How much has been sustained? Modeling Investments(USAID/CSHGP/Save the Children Guinea) What could we achieve if…? Part 2 – Three Recent Examples

13. The Sustainability Framework –Time and Perspective in Evaluation Constraints & Black Swans Improvements continue and/or Achievements are maintained Implement HSS Intervention Stakeholders Adapt & Organize Intervention Ends SF Retrospective: Were gains maintained? Did progress continue? How did system adapt? Prospective: What are the long-term odds based on the results, processes, and systems being built? Source: Taking the Long View: A Practical Guide to Sustainability Planning and Measurement in Community-Oriented Health Programming. 2008. http://www.mchipngo.net/controllers/link.cfc?method=tools_sustain. And: Black Swan, Grey Swan, Sustainability. Or, the difference between Planning and Predicting. Available at: http://cedarscenter.com/resdetail.cfm?resid=125

14. E.g. 1- Measuring sustainability as a programming tool for health sector investments – USAID NepalUsing Composite / Dashboard Measures Compo 1 index Poor Nascent Compo 6 index 2 Intermediate Promising Strong 3 Compo 5 index Compo 4 index Source: Measuring sustainability as a programming tool for health sector investments—report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009 Oct-Dec;24(4):326-50 http://www3.interscience.wiley.com/journal/118903030/abstract

15. Can districts sustain the health of their mothers & children? Kanchanpur and Rasuwa Districts, Nepal - 2006

16. E.g. 2- Sustained Results 5 Years Post-Project. Urban Health / Concern Bangladesh: 1999-2004↨-2009 • From Urban Child Survival Project to Urban Health System Strengthening • Defining a Sustainability Scenario consistent with national policy • Defining ±consistent measures (incl. outcomes and repeated capacity assessments) • Repeated assessments & collective learning • 18 months post-project coaching (minimal) → 3-year and 5-year post-project sustainability assessments Source: Sustainability of the Saidpur and Parbatipur Urban Health Model (Bangladesh) Five Years After the End of Concern’s Child Survival Project . Final Evaluation Report – January 10, 2010. USAID, Concern Worldwide. ICF Macro/. Available at: http://cedarscenter.com/resdetail.cfm?resid=106

17. CWI Bangladesh: 1999-2004↨-2009How Much Sustainability in Health Outcomes? Project Ends

18. E.g.3- Does it matter? HSS at District Level: Sustainability Investments and Child Deaths Averted Save the Children US in Guinea Source: Pro-sustainability choices and child deaths averted: from project experience to investment strategy Health Policy and Planning 2010;1–12. ; doi: 10.1093/heapol/czq042

19. A Simplified Traditional Scenario for Investment Total Investment: $7.5 millions U5 Lives Saved: 2,530 [1,569 – 6,167]

20. Investment Scheme for Maximizing Sustainability – Phase 1

21. Investment Scheme for Maximizing Sustainability – Phase 2

22. Investment Scheme for Maximizing Sustainability – Phase 3

23. Investment Scheme for Maximizing Sustainability – Phase 4

24. Investment Scheme for Maximizing Sustainability – Phase 5 Total Investment: $7.5 millions U5 Lives Saved: 8,485 [4,169 – 8,909]

25. Testable Hypothesis: three-time the impact for the same investment

26. SF 101 - From Planning to M&E Part 3

27. Practical Resource MANUAL www.CedarsCenter.com ANNEXES

29. ‘Sustainability Manual’ Tools • Any-Stage Rapid Tool • Sustainability Planning Checklist • Step 1 • Local system mapping and stakeholder analysis • Visioning activity • Step 2 • From Visioning to Developing a “Sustainability Scenario”

30. Stakeholders’ Analysis for Sustainability Planning

32. ‘Sustainability Manual’ Tools for M&E • Facilitating a detailed planning workshop with local stakeholders • Measurement tools & Environmental Scan • Notes on basing an evaluation on the Sustainability Framework

33. Local actors revise sustainability vision and scenario

34. First Steps in Sustainability Planning– Who Does What?

35. The beauty and dangers of M&e frameworks

36. Example / Community-Based Malaria Project

37. Results Framework Project Objective IR1 IR2 IR3 IR4 Project Accountability Sustainability Framework Elements in the Results Framework Sustainability Scenario Key Elements for Sustainability in the Results Framework of the Project Component 1 Component 2 Component 3 Component 4 Potentially Key Elements for Sustainability in the Larger Environment of the Project and National Stakeholders Shared Accountability Assessing Progress Toward Sustainability

38. Results Framework Aligned to the SF (Taking the Long View. CARE Nepal)

39. The SF “Dashboard”A Picture Is Worth 1,000 Words But Can Sometimes Tell a Lie

40. Conclusion • Some Practical Tool and Guidance; No Insurance Policy • Nothing (Really) New Under the Sun, Except Getting Serious: • Implications for Health and Community Systems • Implications for Scale Up Strategies • Implications for Integration of Health with Sustainable Development in view of Global Challenges • Implications for Evaluation +++ • Don’t Lose Sight of the Forest for the Trees • The [Sustainability] Questions of Donors are also the Questions for Countries Themselves

41. Thank you.