THE GLOBAL FUND PROSPECTIVE COUNTRY EVALUATION

1. THE GLOBAL FUND PROSPECTIVE COUNTRY EVALUATION Synthesis of Findings from 2018Cambodia, Democratic Republic of the Congo, Guatemala, Mozambique, Myanmar, Senegal, Sudan, and Uganda

2. STRUCTURE OF PRESENTATION • Purpose and approach • Methods • PCE focus in 2018 • Key findings and recommendations • Business model • Human rights, key and vulnerable populations, gender • Resilient and sustainable systems for health • Sustainability, co-financing, and transition • Value for money • Added value of the PCE • How are findings being used in PCE countries • Lessons learned • PCE plans for 2019

3. PCE PURPOSE & APPROACH

4. PURPOSE APPROACH Evaluation of the Global Fund business model, investments and contribution to disease program outcomes and impact in eight countries Generation of timely evidence to support programme improvements and accelerate progress towards the objectives of the 2017-2022 Strategy Disease results chains explore links between Global Fund inputs, outputs, programme outcomes, and impact. Programmatic changes to be observed through tracking key indicators. Theory-based approaches and related evaluation frameworks explored how and why Global Fund investments, policies and practices influence disease impact pathways in the results chains

5. METHODS

6. MIXED METHODS • Process evaluation • Document review, meeting observations, key informant interviews, root cause analyses, process mapping • Impact assessment • Existing or modelled secondary data • Analysed retrospective sources to provide context and trends • Quantitative results led to qualitative exploration and vice versa principally through the results chains analyses • Evidence triangulated and strength of evidence ranked across findings

7. FOCUS OF PCE IN 2018

8. Tracked how Global Fund investments translated into activities and programmatic outputs FOCUS OF PCE Building on analysis of funding request and grant making phase in 2017, 2018 focused on early implementation of 34 grants in eight countries, totalling over $2.1 billion in investments during this allocation Identified how the business model enabled and constrained early grant implementation Examined the efficiency and effectiveness of early grant operationalisation

9. PCE Portfolio Characteristics and 2018-2020 Grant Budgets • Sudan (Lower LMI) • Core portfolio • (formerly high impact) • Grants total: $128.4m • COE • Myanmar (Lower LMI) • High impact portfolio • Grants total: $321.5m • Matching funds • Uganda (LI) • High impact portfolio • Grants total: $478 m • AGYW priority country • Intensive support for human rights (all) • Matching funds • CCM evolution • Senegal (LI*) • Core portfolio • Grants total: $73.1 m • Intensive support for human rights (HIV) • Matching funds • Cambodia (Lower LMI*) • High impact portfolio • Grants total: $98.4 m • Guatemala (Upper LMI) • Core portfolio • Grants total: $38.2 m • CCM evolution • DRC (LI) • High impact portfolio • Grants total: $542.9 m • COE • Intensive support for human rights (HIV, TB) • Matching funds • CCM evolution • Mozambique (LI) • High impact portfolio • Grants total: $523m • AGYW priority country • Intensive support for human rights (HIV, TB) • Matching funds • CCM evolution Source: Approved 2018-2020 budgets in $USD. * Income category shifted between 2017 and 2018 Global Fund eligibility lists

10. KEY FINDINGS & RECOMMENDATIONS • Business model • Sustainability, transition andco-financing • Human rights, key and vulnerable populations, gender • Value for money • Resilient and sustainable systems for health

11. KEY FINDINGS & RECOMMENDATIONS • Business Model

12. KEY FINDINGS: Business Model • The Secretariat approved the majority of PCE grants on time • First disbursements (Global Fund to PRs) for the majority of grants were made on time • Approval processes for Matching Funds were aligned with main grants in some cases • Country Teams allowed flexibilities which helped with grant transition • Country Teams played important roles in resolving early bottlenecks Some grant start up processes worked well and as intended

13. FINDINGS: Business Model • Concurrent business model-related processes reduced time and attention from grant start up including for program continuation grants • PR transition created initial implementation delays • Lengthy selection and contracting of implementers, particularly Sub-Recipients by Principal Recipients delayed activity implementation • Some Matching Funds approvals and disbursements were mis-aligned with main grant approvals and this impacted on activity implementation However, some processes worked less well and this affected grant implementation efficiency, contributing to delays and low early absorption rates in most PCE countries

14. Summary of early grant implementation milestones

15. FINDINGS:Business Model Budget absorption for Q1 and Q2 PUDRs 2018 highly variable but low overall: • HIV: 14% • TB: 47% • Malaria: 30% • RSSH: 7% Despite this, our qualitative data suggests that core services (e.g. treatment services provided by national programs) did not stop between grants Q3/Q4 absorption is expected to be higher

16. FINDINGS: Business Model • Reflecting that the provision of core services did not stop between grants, the majority of countries are meeting or nearly meeting performance indicators • HIV: 79% • TB: 96% • Malaria: 80% • This is primarily due to performance indicators being focused on coverage, outcome and impact metrics that relate to the overall national program (rather than grant) performance

17. FINDINGS: Business Model • There is limited/no correlation between financial data on short-term budget absorption and programmatic M&E data • The reliance on imperfect quantitative financial and programmatic M&E data: • Leads to inaccurate grant ratings; • Creates perverse incentives to implementers; and • Poses a risk to Secretariat grant oversight and Board governance functions • While an over-emphasis on absorption for monitoring grant performance has drawbacks, it remains an essential first condition for grant impact. Low early absorption is reflective that budget execution rates should be more carefully anticipated • The lack of reported data on grant-specific outputs (except in some cases) and data at the sub-national level fundamentally compromises analysis and measurement of VfM and could hamper efficient portfolio management

18. RECOMMENDATIONS: Business Model • The Global Fund Secretariat should • Consider flexibilities to the three-year grant cycle to facilitate smoother transition between grants, facilitate early grant implementation and enable adequate time for grant implementation • Update and strengthen guidance for CCMs and PRs on the selection and contracting of SRs to increasingly ‘front load’ PR/SR selection and contracting processes prior to grant implementation. Guidance should include • Metrics which clearly define the time period within which SRs are expected to be selected and contracted by PRs • PRs to work with identified SRs to ensure roles, scope of activities and budgets are agreed during grant making, ahead of the implementation period • PRs should be strongly encouraged to effectively use Pre-Financing Policy flexibilities to facilitate SR preparation (e.g. staff contracting, pre-financing some activities) in advance of grant implementation • Consider embedding matching funds in the timeline for the design, approval and implementation of the main grants to facilitate timely implementation of activities • Consider trying to better link financial and programmatic data by collecting data on the specific outputs achieved through grant implementation, as well as collecting data at the sub-national level, at least for some indicators

19. KEY FINDINGS & RECOMMENDATIONS • Human rights,key and vulnerable populations, gender

20. KEY FINDINGS: Human rights Global Fund interventions to address human rights-related barriers in country grants Activities to reduce human rights-related barriers to services are well represented in HIV grants, but there is less focus in TB and malaria grants

21. KEY FINDINGS: Key and vulnerable populations (KVP) • Investments by disease program largely cover key and vulnerable populations as defined by the Global Fund but many country-defined KVPs do not align with the Global Fund definitions Other Malaria • Pregnant women: Sudan, Uganda • Children <5: Senegal, Sudan, Uganda • Forest workers, dwellers: Cambodia, Myanmar • Seasonal workers: Cambodia, Myanmar • Military, border, armed groups: Myanmar • Camp/settlement occupants: Myanmar Other HIV • Fisher people: Uganda, Senegal • Bridge populations (truckers, security forces, traders, tourism): Senegal • Pregnant women: Senegal • Partners of PLWH: Cambodia • Partners of MSM: Cambodia • Partners of PWID: Cambodia • Clients of FEW: Cambodia • Partners of sex workers: Guatemala • High risk individuals not identifying as KVP: Myanmar Other TB • People living with diabetes: Cambodia, Guatemala, Myanmar, Senegal, Uganda • People living in overcrowded housing / urban slums: Senegal, Uganda • Elderly people: Cambodia, Myanmar • TB contacts: Cambodia, Uganda • Students: Senegal • Health care workers: Myanmar • Urban and rural poor: Myanmar

22. KEY FINDINGS: Gender and Human Rights • Gender and human rights dimensions are not well understood or discussed by stakeholders • Perception that sex-specific targeting alone is sufficient for gender-responsive programming • Lack of experience among Ministry of Health and other stakeholders on gender and on programs to reduce human rights-related barriers to services (including legal dimensions) • Few examples of programs that are actually addressing gender-related vulnerabilities (DRC SASA! pilot project is an exception) • TB and malaria activities are less gender responsive • For example, despite greater TB prevalence in men, most programs lacked interventions that addressed men’s gender-related risks • Overall implementation delays due to sub-contracting issues

23. RECOMMENDATIONS: Human Rights, key and vulnerable populations, gender • The Global Fund Secretariat should • Ensure that Global Fund-supported programs clearly defines key and vulnerable populations, aligned with national epidemiological context and that programs are designed to allow for tracking of progress against key intervention areas (e.g. disaggregation of male/female/trans sex workers, youth, women who inject drugs) • Continue efforts to build in-country capacity and expertise on gender and human-rights related issues, e.g.: • Developing clearer and more accessible guidance on human rights and gender programming and implementation (already underway by Secretariat/CRG); • Ensuring TA is consistent with country needs and facilitating countries seeking technical assistance for reducing gender- and human rights-related barriers (e.g. help the CCM to know that the mechanism exists and see the value in accessing TA to enable stronger more gender responsive planning, implementation, and monitoring) • Country Stakeholders should • More explicitly articulate the gender-related vulnerabilities of men/boys, women/girls, transgender and gender non-conforming individuals, the impact of these on disease-specific outcomes, and specific strategies to mitigate these effects in funding requests and designing disease-specific strategies.

24. RECOMMENDATIONS: Human Rights, key and vulnerable populations, gender • The Global Fund Secretariat and Country Stakeholders should • Encourage more explicit promotion of gender and human rights integration throughout the grant lifecycle, particularly for TB and malaria, including: • Determine the appropriate mechanisms for ensuring the that high quality gender assessments are conducted (or integrated into other assessment practices);  e.g. further direct engagement by Global Fund technical staff in specific country gender assessments • Ensure each CCM has a qualified gender expert engaged throughout the grant design and implementation process with the requirement that the gender expert is fully represented in all processes and decisions • Expand the requirements for addressing gender in funding requests and reporting, using clear guidance that is understandable for both country teams and reviewers • Programming and grant design (e.g. to address social norms, stigma, time use, and intra-household decision-making, not just sex-based targeting) • Implementation (e.g. collection and analysis of programmatic data disaggregated by key populations)

25. KEY FINDINGS & RECOMMENDATIONS • Resilient and sustainable systems for health

26. KEY FINDINGS: RSSH Absorption across RSSH modules during Q1-Q2 2018 was low, in part due to the factors hindering implementation more generally Delays in RSSH implementation are due to similar factors hindering overall grant implementation, such as SR selection and contracting, administrative / logistical hurdles, timing of disbursements, staff turnover, etc. Absorption by RSSH module, by country, across grants with RSSH investments, Q1-Q2 2018

27. KEY FINDINGS: RSSH RSSH coverage indicators predominantly align with HMIS/M&E module, missing an opportunity for monitoring other RSSH priorities Despite limited early absorption, most grants achieved 90+% on RSSH indicators Achievement progress on RSSH HMIS indicators, by PCE country, by grant, Q1-Q2 2018

28. KEY FINDINGS: RSSH Many RSSH investments are considered shorter-term gap investments rather than longer-term investments in more sustainable health system strengthening needs • Cambodia: RSSH investments are mainly targeting “fixable” and “shorter term” issues and lacking strategic focus, rather than tackling systemic “longer term” challenges facing the country, such as human resource capacity • Uganda: Many direct RSSH investments target the malaria program with limited integration efforts (e.g. HRH investments support national malaria control program staff; community response investments to improve malaria component of district epidemic response system) • Myanmar: the HRH investment is largely spent supporting seconded staff on project coordination and logistics related to Global Fund grants, which does little to address the major health workforce shortages facing the country • Sudan: the scattered approach to compiling RSSH activities to fill gaps in short-term disease-specific work plans is thought to have resulted in limited investment in community systems. Stakeholders note this as a missed opportunity for Global Fund contributions to Sudan’s intended national policy shift toward health promotion and primary care

29. RECOMMENDATIONS: RSSH The Global Fund Secretariat should • During the funding request development, consider mechanisms to incentivize stronger alignment of crosscutting RSSH investments to longer-term national strategies for health system strengthening, rather than addressing short-term, disease-specific health system gaps • Improve standardization for categorization of RSSH investments within grant budgets to ensure accurate quantification of Global Fund contributions toward RSSH, including whether simplification is feasible or increased guidance and examples are necessary • Improve monitoring and measurement of the outcomes of RSSH investments, e.g.: • Clear articulation of expected RSSH outcomes, which can be translated into investment guidance, the modular framework and grant performance framework where relevant; • Stronger alignment of grant activities to indicators; and • Consideration (and development of) community systems and responses indicator(s) in the modular framework

30. KEY FINDINGS & RECOMMENDATIONS • Sustainability, transition andco-financing

31. KEY FINDINGS: Sustainability, Transition, Co-financing • All governments of PCE countries have made commitments to meet or exceed Global Fund co-financing requirements • However, external stakeholders (e.g. CSOs, advocates, and evaluators) have been unable to verify fulfillment in a timely manner in most countries • Countries report to the Global Fund on their co-financing and realization of specific commitments periodically, with assurance on country reporting through a variety of country-specific mechanisms • Reporting structures for domestic financing vary between countries, including whether they are accessible to stakeholders • National Health Accounts are one of the primary mechanisms for tracking domestic financing and fulfillment of co-financing commitments, but most mechanisms are retrospective in nature

32. KEY FINDINGS: Sustainability, Transition, Co-financing Even when countries do meet co-financing requirements, PCE countries remain heavily reliant on donor resources to finance the disease program • This may pose a threat to transition readiness, programmatic and financial sustainability • Budget shortfall varies by disease, with HIV typically coming closer to total needs costing than TB • Guatemala is an exception TB 2018-20 Funding Landscape

33. KEY FINDINGS: Sustainability, Transition, Co-financing There is evidence of countries embedding sustainability and transition considerations into program design and implementation • Long-term sustainability planning • Sustainability and transition considerations were integrated into grant design and national strategic plans in Cambodia, Myanmar, Guatemala, and Senegal • Uganda finalized health financing strategy, included a Social Health Protection system and strategic pooling of finances to strengthen purchasing mechanisms • The HIV NSP in Senegal has prioritized the mobilization of domestic resources to sustain the national HIV response • Preparation for transition • Guatemala has embedded a sustainability and transition planinto the TB NSP and is developinga sustainability plan annex to theHIV NSP • A transition readiness assessment in Cambodia found that funding shortfalls in prevention would lead to an increase in HIV incidence and compromise the sustainability of the HIV response

34. RECOMMENDATIONS: Sustainability, Transition, Co-financing The Global Fund Secretariat should • Consider restructuring the country co-financing requirement to more ambitiously increase domestic expenditure on health and the three diseases, with a view to ensuring that domestic financing increases to a level that more fully supports transition and sustainability objectives. Specifically, this might involve: • Expanding upon the co-financing requirement to better reflect the government’s existing financial commitments overall and within the wider health financing landscape (e.g. by setting the co-financing requirement based on more parameters than just the current two) • Increasing the minimum level of co-financing that is acceptable to the Global Fund (i.e. increasing the co-financing requirement but not necessarily the co-financing incentive) • Strengthening the incentive for countries to increase domestic expenditure on health and the three diseases by making additional resources available to countries that invest above the minimum acceptable level of co-financing (via a separate mechanism than the existing incentive, which can only be taken away)

35. KEY FINDINGS & RECOMMENDATIONS • Value for • money

36. KEY FINDINGS: Value for money: Economy Analysis of PQR data suggests that economy has improved over time across the grants, with health commodity prices falling, often below global reference prices Some unit costs used for budgeting do not closely reflect actual costs, potentially leading to: • Global Fund paying above the lowest possible cost for inputs; and or • Low budget absorption

37. KEY FINDINGS: Value for money: Efficiency & Effectiveness Strong examples of efforts to improve efficiency of grant design and national programs, particularly in countries facing significant reductions in program budgets Program management costs vary significantly across countries and by type of PR, with substantially higher costs for UN agencies and CSOs than for governments Cost-effectiveness considerations inform program design and decision making in most settings (such as through modelling) but not systematically

38. KEY FINDINGS: Value for money: Equity • While equity is often discussed, trade-offs between equity, cost-effectiveness and programmatic targets are dealt with differently (often informally) • More could be done to ensure that Global Fund-supported activities (and their benefits) are fairly distributed amongst target recipients, e.g.: • A review of the gender-related dimensions of equitable access to services and reduction of vulnerabilities for men, women, boys, girls, and transgender individuals • Some evidence that over ambitious target setting vis-à-vis available funding has been counterproductive to the prioritization of hard-to-reach areas • Despite some examples of Global Fund support being used to reduce financial barriers to accessing services, this still poses a significant issue

39. RECOMMENDATIONS: Value for money • The Global Fund Secretariat, together with partners, should • Expedite work to collect unit/service delivery costs at the country level and use this as a basis for budgeting, with clear guidance on appropriate formulae to inflate estimates to allow for inflation, price changes, currency shifts, etc. • Consider ways to strengthen country-level and/or grant-specific analysis of VfM throughout the grant life-cycle (while considering the burden of reporting), such as by: • Collecting and analysing grant-specific output data • Extending reporting tools to collect sub-national data • Creating performance targets that better address equity considerations • Requesting that PRs/countries report against trends in efficiency and effectiveness

40. ADDED VALUE OF PCE

41. Added value of the PCE: Global level PCE analysis provides in-depth knowledge of the complexities of grant implementation unlikely to be found in thematic reviews/short country visits PCE synthesis represents a whole that is greater than the sum of the parts with recommendations derived from and consistent with the evidence from multiple countries PCE evidence is informing and/or validating findings from TERG Thematic and Strategic Reviews (e.g. RSSH and Partnerships) and is able to use other TERG Review findings prospectively PCE findings on lessons learned for key processes (e.g. funding request development, SR selection, etc.) will inform Secretariat planning of the next implementation cycle PCE is able to respond to emerging TERG or Secretariat issues (e.g. business model issues, CRG needs) and rapid requests e.g. feasibility of implementing new MDR-TB treatment guidelines (see example in next slide)

42. The PCE platforms in Myanmar, Sudan and Uganda were mobilized at short notice by the TERG to facilitate a preliminary review of the feasibility and bottlenecks for transitioning to and implementing the new guidelines (Dec 2018 start -Jan 2019 end) Added Value of the PCE Conducting a rapid review on the feasibility of implementing new treatment guidelines for MDR/RR-TB and LTBI in three PCE countries High levels of trust between PCE team and country stakeholders enabled access to key informants, documents and data and this contributed to the review’s success The PCE review’s findings have provided a baseline for the PCEs to follow up prospectively in Myanmar, Uganda and Sudan The PCE review provided country recommendations for programmatic course correction and Global Fund recommendations for the current and next grant cycle The PCE platforms are developing capacity and agility to respond to emerging issues and have the potential to be further development and utilized. Discussions are underway for further TB-related findings to be generated through PCE in this way

43. Added value of the PCE: Country level Targeting PCE findings to national program managers: The ability to disseminate emerging findings in a timely manner is a core strength of prospective evaluations and provides an opportunity for the PCE to contribute to continuous quality improvement Opportunities for subnational data collection and analysis can add value to national level perspectives Synthesis adds value at country level, enabling stakeholders to compare their responses to those of other countries as well as understanding how the PCE is part of a larger strategic process Country stakeholders’ appreciation for documenting the challenges, successes, and learnings throughout the Global Fund grant cycle – some of which are previously known, but not systematically or independently documented, nor synthesized across countries

44. What have we learned from the PCE approach? Platform/Methods • Results chain is helpful analytic tool • PCE knowledge of Global Fund takes time to develop but now seeing capacity & agility to respond to emerging issues • Balancing competing priorities and multiple stakeholders is challenging • Difficulty with timely feedback when evaluating processes that happen once during the grant cycle – findings relevant in 3 years PCE Team Structure • Strong linkages between global and country evaluation partners is essential • Various staffing models among GEP and CEP – but tracking 3 diseases requires sufficient people for embedded evaluation model • Opportunities for cross GEP/CEP learning: in-person, webinars, TERG meetings • Relationship building with country stakeholders is critical PCE Reporting/Dissemination • Dissemination needs to be aligned with critical data use periods • Annual report deliverable may be inconsistent with stakeholder preferences; shorter, more frequent briefs likely to be better • PCE teams lack knowledge translation expertise – this could help in dissemination findings and strengthening feedback loop PCE / TERG / Country Team Engagement • TERG meetings and presentations require significant time and input (high transaction costs) • Some inconsistencies in TERG feedback over time • CT engagement early and often is critical to ensure PCE is helpful to CT’s work • Stronger engagement with Global Fund Secretariat could help ensure added value and synergies, while avoiding duplication

45. Plans for the PCE in 2019 Process • Continued grant implementation monitoring and business model process tracking • Greater use of root cause analysis to understand implementation barriers and facilitators • “Deeper dive” inquiries into linkages between activities and outcomes along the result chains to help explain observed trends, using thematic areas as possible analytic lenses • Stronger emphasis on timely feedback to country stakeholders and use of PCE findings

46. Plans for the PCE in 2019 Impact • Differentiated approach by country and disease • Extend analysis of results chains • Additional indicators and paths • Country-specific tailored analysis • Programs, populations or geographic regions of specific interest to the country • Model-based impact analysis • Statistical correlations between adjacent elements of results chains (i.e. inputs vs. outputs; outputs vs. coverage, etc.) • Structural equation modeling where complete data at sub-national level are available • Alternative (e.g. causal inference; epidemic) models where less complete data are available

47. Example results chain for malaria