Evaluation of Accelerated Child Survival Programs in Africa

1. Institute for International Programs ASADI V Accra, November 2009 Evaluation of Accelerated Child Survival Programs in Africa Jennifer Bryce Institute for International Programs The Johns Hopkins University

2. Outline – What have we learned? • From the evaluation of the Accelerating Child Survival and Development (ACSD) Program? • From prospective evaluations of the Catalytic Initiative to Save One Miiilion Lives (CI) to date? • The way forward

3. ACSD, 2002-2005 11 countries in Africa Support from CIDA and other partners Implemented through UNICEF Aim: To reduce mortality among children less than 5 years of age Strategy: Accelerate coverage with three packages of high-impact interventions, with a special focus on community-based delivery

4. ACSD Program:Intervention packages EPI+ Vaccinations Vitamin A supplementation ITNs for U5s & pregnant women De-worming • IMCI+ • Facility IMCI • Community case management (CCM) of childhood illnesses • Diarrhea: oral rehydration therapy (ORT) • Malaria: based on current policy • Pneumonia: referral to facility • Promotion of timely initiation of breastfeeding, exclusive breastfeeding to 6 months, timely complementary feeding • Promotion of household consumption of iodized salt • ANC+ • Malaria prevention in pregnant women (IPTp) • Tetanus Toxoid • Iron/folic acid supplementation • Vitamin A post-partum • PMTCT

5. The retrospective independent evaluation of ACSD • High-impact districts in Benin, Ghana, Mali • Standard indicators • Existing DHS/MICS with oversampling • National comparison areas • Documentation of program implementation & contextual factors • No cost component • Stepwise design

6. ACSD Implementation: GHANA Key: Bars represent districts in the following order: Builsa, Bawku East, Kasena-Nankana, Bolgatanga, Bawku West, Bongo

7. Key Coverage for EPI+ interventionsbefore and after ACSD, in HIDs Before ACSD After ACSD Mali Benin Ghana * * 3 * * * * * * * * Increases in coverage across the board in Ghana and Mali; Benin achieved increases for vitamin A and ITNs. * Change was significant at p < 0.05.

8. Key Coverage for IMCI+ interventionsbefore and after ACSD, in HIDs Before ACSD After ACSD Mali Benin Ghana * * * * * * * No coverage gains, and some significant losses, in sick child care. Exclusive breastfeeding increased in Ghana, declined in Mali. * Change was significant at p < 0.05.

9. Key Coverage for ANC+ interventionsbefore and after ACSD, in HIDs Before ACSD After ACSD Mali Benin Ghana * * ** * * * * * * * * * Ghana and Mali improved care for childbearing women; delivery of TT and postnatal vit A benefited from EPI system in Mali. *Change was significant at p ≤ 0.05. ** Measured level was 28%, but country team reported this was incorrect as IPTp had not been implemented in 2001.

10. Under-five mortality in the ACSD HIDs 19% (p=0.10)

11. Under-five mortality in the ACSD HIDs and national comparison areas Declines in U5M in ACSD focus districts, but not greater than national comparison areas. No changes in child nutritional status attributable to ACSD.

12. Did ACSD implementation contribute to reducing inequities? Socioeconomic inequalities, showing breakdown by wealth quintiles of ANC 3+ coverage in ACSD “high-impact” zones and the comparison area, Mali, 2006-7. • Yes, in Mali, where socioeconomic and urban/rural inequities decreased more in the ACSD HIDs than in the comparison area. • Baseline sample sizes too small to support analysis of equity trends in Benin or Ghana.

13. Conclusions & implications • Intervention coverage CAN be accelerated if there is adequate funding & human resources. • Acceleration of mortality declines require: • Focus on interventions that have a large and rapid impact on major causes of child death • Sufficient time to fully implement approach and for coverage to translate into declines in mortality • Reasonable expectations, given level of resources ► Work for closer match between program resources & cause of death ►Be realistic about what can be accomplished ► Level of funding matters

14. Conclusions & implications • Policy barriers prevented key ACSD interventions directed at pneumonia and malaria from being fully implemented. • Breakdowns in commodities and gaps in funding stall progress toward impact. • More attention and operations research needed on incentives and supports for community-based workers ► Work for policy reform as first step, where needed ► Pay attention to health systems supports such as commodities, supervision, & incentives

15. Contributors & acknowledgements Contributors Acknowledgements Ministries of Health, National Statistics Offices, UNICEF country staff, Collaborators in documentation UNICEF regional and global staff Genevieve Begkoyian, Mark Young, Sam Bickel Technical consultants Trevor Croft, Macro International UNICEF leadership For their commitment to learning and change Jennifer Bryce Kate Gilroy Elizabeth Hazel Gareth Jones Robert Black Cesar Victora

16. Part 2 Evaluating The Catalytic InitiativeTo save a million lives

17. Two Linked Evaluations The Catalytic Initiative Independent Evaluation of the MNCH Rapid Scale-Up “Real-time” Mortality Monitoring (RMM) Overall objective: To monitor changes in under-five mortality in real-time. • Overall objective: Provide “proof of concept” that proven interventions can be scaled up rapidly to reduce newborn and child mortality. • Supported by: BMGF • Implementing partners: Governments and UNICEF, WHO, UNFPA • Supported by: CIDA • Implementing partner: Governments and UNICEF • Countries: Burkina Faso, Malawi, Mozambique • Countries: Ghana, Malawi, Mali, Mozambique

18. Process of evaluation design

19. Progress: MalawiIn-country partners: Centre for Social Research and National Statistics Office Implementation Evaluation (full) Mortality monitored by: Having CHWs report vital events Calibrating facility deaths against community deaths Two rapid survey approaches Full documentation of program & contextual factors Quality of care assessments at 1st-level facilities and for CHWs Costs & equity tracked 12 districts: 6 “accelerated” and 6 routine National platform approach under discussion • Features of accelerated approach: • Government-paid CHWs trained to deliver CCM for pneumonia, malaria, diarrhea (including zinc) • Strengthening district health management • Implementation status: • In 10 intervention districts, 5-15% of CHWs trained by June 2009

20. Progress: MozambiqueIn-country partner: Eduardo Mondlane University Implementation Evaluation (full) • Mortality monitoring by calibrating facility to community deaths. • Stepped-wedge design based on scheduled cohorts for Rapid Scale Up • Documentation of baseline health & nutrition, inputs & contextual factors and coverage for each cohort using national evaluation platform approach • Will support dose (program intensity) – response (coverage & modeled impact) analyses • Features of accelerated approach: • Increased access to quality child health care in facilities • Quality of immunization services improved • Long-lasting insecticide-treated nets (ITNs) distributed and used • Vitamin A supplementation • Breastfeeding promotion • BMGF funds used to fill gaps in maternal health • Implementation status: • CI planned to be implemented in 33 districts each year from 2008 to 2012

21. Progress: Burkina FasoIn-Country Partner: Institut Supérieur des Sciences de la Population (ISSP) Implementation Evaluation (no RMM) • National platform approach in 9 intervention and 2-3 comparison districts • New “LiST” survey to collect district-level estimates of coverage for proven MNCH interventions • Modeled mortality using LiST • Analysis using pre-/post-intervention with comparison and/or dose response • Features of accelerated approach: • Volunteer community health workers providing: • CCM for diarrhea (ORT + zinc) and malaria (ACT) in 9 districts • CCM for pneumonia in 2 districts • Strengthened district planning and supervision • Implementation status: • Materials ready; cascade training of all CHWs in 9 districts to be completed before end 2009

22. Lesson 1:Existing plans include high-impact interventions… *If targets fully achieved at adequate service quality.

23. …but if feasibility and speed are issues,just 4 or 5 interventions can achieve ≥ 20% reduction in U5M by 2015 Pre-publication results; not for citation or distribution

24. Lesson 2:Implementation takes time Implementation status of functional village health clinics with CHW trained in CCM, Malawi CI districts, June 2009 (18 months after project start-up) *1 trained CHW per village health clinic

25. …especially when policy reform is needed. In Mali, the MoH scheduled a “forum” to decide on CCM for childhood pneumonia & malaria. Discussions about how to implement are still under way Forum held; agreed “YES” on CCM Original date (cancelled) Planned (cancelled) Planned (cancelled) July 2008 March 2009 November 2008 February 2009 November 2009 Months 4 + 3 + 1 = 7 months in a 3-year CI project

26. Lesson 3:“Virgin” comparison areas do not exist • Simultaneous implementation of multiple programs • Separate, uncoordinated, inefficient evaluations, if any Mozambique

27. Lesson 4: There are no shortcuts for mortality measurement (at least not yet) • Capturing a 25% difference-in-difference for rates of child mortality in a two-year period requires a survey of ≈ 12,300 households in each group* • Promises of measuring declines in 1 year using survival analysis or other techniques still require these prohibitively large sample sizes, plus detailed info on age of death • CI work on “real-time” mortality monitoring will assess the validity of alternative methods, but in first trials require validation against a gold standard *based on Malawi; sample sizes will increase as mortality rates decrease, e.g. in Ghana

28. RMM Options by country

29. Contributors In-country partners IIP-JHU Burkina Faso: ISSP, INSP Ghana: Noguchi Institute, University of Ghana Malawi: NSO, CSR, Department of Economics, University of Malawi Mali: CREDOS Mozambique: Eduardo Mondlane University Agbessi Amouzou, Abdullah Baqui, Robert Black, Jennifer Bryce, Kate Gilroy, Elizabeth Hazel, Gareth Jones, Marjorie Opuni, Jeremy Schiefen, Cesar Victora, Damian Walker

30. Part 3 The way forward:National EvaluatIon Platforms (NEPs)

31. What is a national evaluation platform (NEP)? • District-level databases covering the entire country • Containing standard information on: • Inputs (partners, programs, budget allocations, infrastructure) • Processes/outputs (DHMT plans, ongoing training, supervision, campaigns, community participation, financing schemes such as conditional cash transfers) • Outcomes (availability of commodities, quality of care measures, human resources, coverage) • Impact (mortality, nutritional status) • Contextual factors (demographics, poverty, migration) • Permits national-level evaluations of multiple simultaneous programs

32. NEPs: A common evaluation framework • Common principles (with IHP+, Countdown, etc.) • Standard indicators • Broad acceptance

33. NEPs: Sound evaluation principles • In-country evaluation counterparts • Local expertise, able to provide continuing evaluation research support to the MOH • Continuity of inputs from evaluation team; cross-country network of investigators • Linked “independence” • Investigators not involved in implementation of MNCH activities • Regular exchange with in-country implementation team • Ongoing activity; not one-off approach • Attribution by approach • Documentation of all contributions • Comparison of accelerated approach with “routine” approach

34. What types of questions can an NEP answer? • Are programs being deployed where need is greatest? • Is implementation strong enough to have an impact? • Did programs increase coverage? • Was coverage associated with impact? • How equitable are the programs? • How much did programs cost?

35. How can the MOH and partnersuse the platform? • Which approaches or combinations are contributing to rapid scale-up? • Are some districts more efficient than others? Why? • Are changes in epidemiology (e.g., due to IRS) reflected in reallocation of resources in district plans? To learn from well-performing districts and guide those doing less well

36. Why should you consider a national platform approach (or not) ? Advantages Limitations Observational design (but no other alternative may be possible) Cost, particularly due to large size of surveys (!But cheaper than many standalone surveys!) Requires transparency and collaboration by multiple programs and agencies • Adapted to current reality of multiple simultaneous programs/interventions and partners • Flexible design allows for changes in implementation • Can be used to evaluate multiple programs (child survival, HIV, malaria, maternal health, etc.) • Supports country ownership and capacity building

37. Thank you Further details at www.jhsph.edu/iip and www.cherg.org