Assessing Sustainability in Global and Regional Partnership Programs

1. Assessing Sustainability in Global and Regional Partnership Programs Elaine Wee Ling Ooi AEA Conference November 13, 2009

2. Main Messages • Program Sustainability • Go beyond assessing program sustainability to assessing sustainability of outcomes • Partners and Stakeholders • A GRPP is only one player among a cast of several • Sustainability of outcomes depend on the individual contributions of other development actors, and national/local stakeholders • Replicability, Devolution & Exit • These achievements need to be assessed, depending on GRPP maturity/type

3. Special Features of GRPPs in Relation to Sustainability • Programs do not start off with predetermined end-points; they may modify business model over time if needed to improve on performance. • Programs are typically grant-financed, with little capacity to generate revenue from their own resources. • Outcomes and their sustainability depend on the complementary activities of partners, and the capacity of country-level stakeholders. • Many programs have phased implementation plans. • Long-term sustainability may require devolution or exit from regional/country-level activities.

4. Assessing Sustainability – Suggested Approach • Theory of sustainability • Sustainability of the program • Complementary activities of development partners • Capacity of country-level stakeholders • Scaling up and replicability • Devolution and/or exit

5. Theory of Sustainability • Sustainability is the continuation of benefits arising from a program’s activities after the activities have been completed. • Essential evaluation question: What is the probability or resilience to risk of continued benefit flows over time? • Two sub-questions: • What are the requirements – in theory – to sustain key outcomes of the program in the future? • To what extent are the program and its partners – in practice – paying attention to these requirements in the present? • Answers will depend on • Types of activities that the program is undertaking – whether networking, technical assistance, or financing global, regional, or national public goods • Maturity of the Program.

6. Good Practice Example: Sustainability of the Program Roll Back Malaria (RBM) • Global Coordinator of Malaria Control; also provides TA • Waning interest from core donors affected planned ‘roll out’ activities • Examined future of partnership • Conducted “partner value mapping exercise”, including partners’ expectations of RBM role and functioning • Had founding members “adapted” business plans to further RBM goals? • Contrasted RBM structure with other partnerships. Was RBM-WHO host relationship optimal to fulfilling mandate? • Financial health; resource mobilization capability of RBM • Partnership recharged; formal Governing Body created; enabled RBM to position itself for operational expansion

7. Good Practice Example: Partners’ Complementary Activities at Country Level UNAIDS • Global Coordinator for HIV/AIDS; strategic TA • Examined contributions of partners’ complementary programs: • Used a Web-based UN Partnership Assessment tool • Assessed effects of interdependency, over-reliance, coordination, and synergies on overall UNAIDS achievements • Also assessed working relationship with PEPFAR, Gates, civil society, and private sector in terms of policy and TA development • Evaluations have done a good job of assessing effectiveness of partners’ activities and increasingly their sustainability

8. Good Practice Example: Capacity of Country-Level Stakeholders African Management Services Company (AMSCO) • Provides expatriate CEOs and management training to help SMEs become profitable and sustainable • Evaluation assessed: • Financial Sustainability of AMSCO – self supporting, deriving income for services rendered to companies • Commercial Viability of SMEs using indicators such as profitability, management stability, business and operational changes; and compliance with environmental, health, and safety standards • Sustainability of Benefits: Market for expatriate CEOs and management training and establishment of other management consulting firms. • Easier to do such an assessment in older programs with more narrowly defined goals and targets

9. Good Practice Example: Scaling Up and Replicability Millennium Villages Program (MVP) • Piloted an “integrated package” of tried and tested low-cost interventions to bring rural communities out of poverty • Evaluation purpose: To assess “proof of concept” and learning: • Factors necessary for sustainability within and scalability beyond pilot sites • Factors considered: • Funding; absorptive capacity; cost-effectiveness; preparedness for phase 1; • Alignment with national and local priorities. • Addressed “proper sequencing” of interventions for “scaling up” • Replicate first measures least in need of reform or adaptation • Second order measures: Simplification, adaptation and contextual fit • Consider local expectations about targets; Contextual fit important

10. Good Practice Example: Devolution/Exit African Program on Onchocerciasis (APOC) • Establishing sustainable community directed treatment (CDTI) systems for riverblindness in 19 countries • Evaluation assessed regional- and local-level sustainability • Desirability of continuing partnership after 2010 to coordinate and sustain essential trans-boundary activities (surveillance & monitoring) • APOC instruments, structures and procedures for devolution to countries and communities • Evidence of institutional, political and budget support by countries, communities and NGOs – “sustainability action plans”; results of “independent performance monitoring” and integration with local health systems • Criteria for devolution: predefined, agreed upon by APOC partners, beneficiaries and evaluation team

11. Major Conclusions • Beyond Program Sustainability: • Evaluation can more concretely measure impact and benefits in older programs, especially those with well defined targets and goals • Partners and Stakeholders: • Very mature programs with strong country-level activities more cognizant of complementary contributions by other partners. • Contextual fit important – strongly consider local expectations about targets in assessments. • Replicability and Devolution: • Early planning/agreement necessary: Criteria for scaling up and devolution need predefinition; and agreed upon by partners, beneficiaries and evaluators

12. Thank You