Framework for evaluation of scaling up of programmes

1. Framework for evaluation of scaling up of programmes Consultative workshop on Health Impact Accounts Brussels 6 February 2012

2. Demand for health impact evaluation • MDG 2015: accurate measurement of child and maternal mortality • GAVI: immunization coverage  immunity  incidence and mortality reductions • Global Fund: coverage of interventions (ART, bednets and malaria treatment, TB treatment), impact on morbidity & mortality • PEPFAR: new infections (children, adults), mortality • NCD political declaration: targets on mortality due to major chronic diseases, risk factor prevalence • Commission on Information & Accountability for Women's and Children's Health: birth and death registration • Post 2015 development agenda: place of health as intervention; health as an outcome measure

3. Accountability Monitor GLOBAL GLOBAL Monitor Review COUNTRY Action Action Review

4. Evaluation = Accountability? • Accountability: was the money well spent? • Evaluation also means learning: which components of the programmes and interventions worked best? what are the implications for future programs? • Current demand for results and relating those to investments and interventions: • global level: lives saved computations • large-scale effectiveness evaluation at country level • Supply: • Somewhat more investment in evaluation • More retrospective and to a lesser extent prospective evaluations

5. Evaluation of effectiveness of large-scale programmes • More surveys to ascertain health impact • India: conditional cash transfer program for facility births, using District Level Household Surveys (Lim et al., 2010): coverage, not maternal mortality • West Africa: child survival program (Bryce et al. 2010): used districts, DHS • Indonesia: World Bank MCH interventions (Baird et al., 2011): provinces 15 yr • Issues: impact measurement, measurement of intervention implementation strength, attribution, no counterfactual, time lag • Large scale evaluations of complex interventions • Global Fund five year evaluation: retrospective, multiple diseases, multiple data sources, multiple countries, short time period, heavy reliance on modelling, generated no major new evidence of impact • Avahan HIV prevention: $27 mln investment in evaluation, prospective (2003), multiple data sources; dozens of publications (STI, BMC Public Health); lives saved based on money disbursed and HIV antenatal surveillance in districts (Ng et al., 2011): 100,000 in 5 years (25,000-207,000) • GAVI: prospective evaluation investment decision board

6. Country accountability processes in context of National Health Strategies

7. Country-led platform for information & accountabilityMonitoring & review of the national health strategy External validation and estimates Common standards and tools Monitoring reports GAVI reporting MDG / UN reporting Country data generation & compilation Analysis & synthesis Country data information generation & compilation Global Fund reporting Evaluation Reviews Programme reports Statistical reports Reviews Data quality assessment Independent reviews PEPFAR reporting Programme Reporting (TB, MCH, HIV, etc.) Minimization of reporting requirements Harmonization of reporting requirements

8. Inputs & processes Outputs Outcomes Impact Improved health outcomes & equity Social and financial risk protection Responsiveness Efficiency Intervention access & services readiness Intervention quality, safety Governance Financing Coverage of interventions Prevalence risk behaviours & factors Infrastructure; ICT Health workforce Supply chain Information Indicator domains Data collection Population-based surveys Coverage, health status, equity, risk protection, responsiveness Facility assessments Service readiness Administrative sources Financial tracking system; NHA Databases and records: HR, infrastructure, medicines etc. Policy data Clinical reporting systems quality, coverage, health status Civil registration Analysis & synthesis Communication & use Technical framework for M&E and review of National health strategies Data quality assessment; Synthesis of progress and performance; Evaluation Regular country health sector review processes; Global reporting

9. National evaluation platform (Victora et al. 2010) • Traditional evaluation designs do not work for large scale effectiveness evaluation: no comparison population, counterfactual difficult, ethical reasons • Prospective design, linked with national monitoring efforts • Use districts as the unit of design and analysis • Contextual information • Ensure continuous monitoring of input, process and output indicators • Collect additional data before during and after the evaluation period using multiple methods • Baseline (and intermediate) survey, coverage, adequate sample size for district analysis • End survey: include retrospective mortality data (expand planned surveys such as DHS if possible) • Use multiple data analysis techniques to deal with data gaps and biases • Areas with and without programmes (before-after analysis with comparison group); stepped wedge design (if sequential roll-out); dose response analyses • Include interim (formative) and summative analyses • Promote country ownership, transparency and donor coordination, while maintaining a rigorous of the cost effectiveness of different scale-up approaches Source: Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the MDG era and beyond: A new approach to large-scale effectiveness evaluations. Lancet, published on line 9 July 2010.

10. Inputs & processes Outputs Outcomes Impact Intervention access & services readiness Intervention quality, safety and efficiency Coverage of interventions Prevalence risk behaviours & factors Improved health outcomes & equity Financial risk protection Responsiveness Infrastructure / ICT Health workforce Supply chain Information Financing Governance Comprehensive analysis of country health progress and performance A stepwise approach Have finances been disbursed? Have policies been changed? Contextual changes Non health system determinants Is the process of implementation happening as planned? Has access to services improved? Did the quality of services improve? Has utilization improved? Did intervention coverage improve? Have risk behaviours improved? Contextual changes Non health system determinants Have health outcomes and equity improved? Are services responsive to the needs? Are people protected against financial risks?

11. Evaluation platform Advantages Adapted to current reality of multiple simultaneous programs/interventions Integrated with country health system Promotes country ownership and donor coordination Evaluation as a continuous process Flexible design allows for changes in implementation Limitations Observational design (but no other alternative is possible) High cost particularly due to large size of surveys Requires transparency and collaboration by multiple programs and agencies

12. Global evaluation framework • Evaluation = accountability + learning • Global computations only as good as country data inputs • Critically dependent on investments in country platform for information & accountability • Data availability (e.g. causes of death) and quality (e.g. program implementation strength) are key; (but more data may also imply more complicated answers) • Risk of disconnect between global and country relevance: • global health impact accounts not relevant for countries • Computations likely to be complex and full of assumptions • Countries do not care about attribution to development partners