Evaluating the system-wide effects of HIV scale-up: methodological gaps, challenges and recommendations

1. Evaluating the system-wide effects of HIV scale-up: methodological gaps, challenges and recommendations David HotchkissHealth Systems 20/20/Tulane University

2. Introduction • Wide agreement that further evidence is needed on how GHIs have influenced health systems • Research in this area has been characterized as serendipitous rather than systematic • Black box between inputs and outcomes, and intended and unintended effects, still needs illumination

3. Overview • What type of evidence is needed? • What are the obstacles to evaluation? • What are the available M&E approaches? • Where do we go from here?

4. What type of evidence is needed? • Routine monitoring • Monitoring of global goals and GHIs • Country-level health system and program monitoring focused on inputs, processes and outputs • Special purpose evaluations of the impact of health systems strengthening initiatives • All these aspects are overlapping and require an integrated approach at the country and global levels

5. What are the obstacles to evaluation? • Evaluations require collective action • Systems research traditionally has not received adequate financial support • High quality studies “do get done, but not in the numbers or with the quality that are justified by the global benefits” (CGD 2006) • Arrangements for prospective HSS assessments typically not built in from the beginning • Too little investment in baseline survey data • No treatment and control groups to assess counterfactual

6. What are the obstacles to evaluation? (2) • Routine tracking data should provide much of the data required for evaluation, but unfortunately, RHIS data are often incomplete and of poor quality • Attribution is challenging from a technical perspective • Systems strengthening a long-term, complex process • GHIs are also extremely complex and dynamic • Difficulty in attributing impact to any one GHI • Other factors, including concern about possible unfavorable results and limited capacity

7. What approaches and methods are available for monitoring and evaluation?

8. Example of a conceptual framework: IHP+ common M&E framework

9. What approaches and methods are available for monitoring? • National Health Accounts and Sub-Accounts, a tool for tracking sources and uses of funds • Comparison of NHA data over time can yield useful insights on system-wide effects • Health Systems Assessment Approach: A How-To Manual (Health Systems 20/20) • Provides indicators and sources of data by WHO’s health system functions

10. What approaches and methods are available for evaluation? • Retrospective • Global cross-country comparative analysis • Country-level mixed methods analysis (quantitative and qualitative, including case studies) • Prospective • Experimental designs assessing impact of interventions; economic evaluations of costs and benefits • Consider intervention-specific approaches • For some types of interventions, such as financing, service delivery, rigorous methods are available • For others, such as RHIS, more work is needed

11. Where do we go from here? • Great opportunity exists to invest in systems research • GFATM and GAVI increasing systems funding • G8 placing more emphasis on systems strengthening • IHP+ has developed a common evaluation approach • Develop “a new field of science for health outcomes and systems research among a constituency that has been fragmented in the past” (MPS 2008)

12. What does this involve? • Advocate rigorous M&E as a key component of the health systems strengthening agenda • In establishing HSS objectives of GHIs, establish systems to monitor and evaluate process • Align health systems M&E with implementation, country planning cycles and mechanisms • Continue to develop frameworks, metrics and methods • Invest in RHIS to strengthen data quality and use • Develop intervention-specific frameworks and methods • Continue to develop mixed-methods approaches • Look for opportunities for prospective evaluations

13. Thank you!

14. Selected references Atun RA, Bennett S, Duran A. When do Vertical (Stand-Alone) Programmes Have a Place in Health Systems? Policy Brief, WHO European Ministerial Conference on Health Systems, 25-27 June, 2008, Tallinn, Estonia. Banteyerga, H, Kidanu, A, Stillman, K. (2006). The Systemwide Effects of the Global Fund in Ethiopia: Final Study Report. Bethesda, MD: PHRplus. Abt Associates Inc. Global HIV/AIDS Initiative Network (GHIN) (2006). A Generic Guide to Research Practice: Following discussion at Lilongwe workshop of GHIN African teams. International Health Partnership (2008) Monitoring Performance and Evaluating Progress in the Scale Up for Better Health: A Proposed Common Framework. M&E Working Group. Kruck ME, Freedman LP (2008) Assessing Health System Performance in Developing Countries: A Review of the Literature. Health Policy 85. Rockefeller Foundation (2008) Leveraging HIV Scale-up to Strengthen Health Systems. Report of high-level meeting in Bellagio, Italy, 2-5 September WHO (2007) Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes. WHO, 2007.

15. Selected references (2) WHO (2008) Maximizing Positive Synergies Between Health Systems and Global Health Initiatives. Report on the 3rd expert consultation, WHO, Geneva, 2-3 October. WHO Maximizing Positive Synergies Collaborative Group (2009) An Assessment of Interactions Between Global Health Initiatives and Country Health Systems. The Lancet (373) June 20.