Measuring success for mHealth Lessons from monitoring and evaluation of Vodafone Foundation & UN Foundation’s mHeal

1. Measuring success for mHealthLessons from monitoring and evaluation of Vodafone Foundation & UN Foundation’smHealth program in Africa 28 October 2009 Andrew Stern – Partner, Dalberg Global Development Advisors

2. Background and summary • Dalberg has been working with UNF and Vodafone Foundation for two years to understand the impact of its mHealth program, with an in-depth focus on the programs in Senegal, Ethiopia, and Kenya • The approach used a ‘Theory of Change’ approach to analyze and monitor activities, outputs, and outcomes that lead to impact • Focusing on activities, outputs, and outcomes for monitoring & evaluation enables: • Identification of critical constraints to achieving impact and therefore an opportunity to develop mitigation approaches; • Focus on cost efficiency to yield greater program effectiveness; • Demonstration of relative effectiveness of a program across geographies • Comparing across countries, major success drivers have been identified including: • The quality of data-driven analytical and decision-making processes to make use of the data being collected; • The fact that narrow but deep pilots often work better for demonstration; and, • How critical it is to ensure complementary inputs, such as transportation costs, to ensure collection of data

3. The UNF-VF partnership Since 2005, the Partnership has worked with DataDyne and WHO to develop and roll out EpiSurveyor EpiSurveyor is now active in 19 countries Objective: Bring the latest technology solutions to UN field work, using sustainable, digital tools to save lives Solution: EpiSurveyor enables development of customizable surveys for cell phones and PDAs Partners: DataDyne, World Health Organization, Ministries of Health, UNF-VF Duration: 2005-2010

4. Theory of change Impact ~3-5 years Saving and improving lives Outcome ~6 months • Improved health systems and monitoring • More effective campaigns (vaccinations, etc.) • More effective outbreak response Outputs ~3 months Health data collection Analysis and information sharing Decision-making and action to meet needs National Regional District Activities ~3 months Collection management and supervision Equipment Software Development Training Logistical support Customizable survey development Partnership activities 3

5. Indicators and performance management Intent Indicator Impact ~3-5 years Reduce lives lost and improve lives 10. Disability adjusted life years (DALYs) prevented Outcome ~6 months Country health systems with more, better, and faster data for decisions and action 9. Percent of survey questions showing positive improvements from prior month 8. Percent of decision-making meetings that discuss the data 7. Records analyzed as % of records collected 6. Records collected as a % of visits conducted 5. Visits conducted as % of potential visits in program 4. Visits covered by program as % of all potential visits 3. % of Ministry of Health divisions using PDA Outputs ~3 months Provide data collection method for as many health applications and health facilities as possible Activities ~3 months Do more with same funding, or less 1. Cost per survey record collected 4

6. Focusing on measuring outputs and outcome allows for the identification and mitigation of critical challenges that constrain impact Outcome ILLUSTRATIVE • Improved health systems and monitoring • More effective campaigns (vaccinations, etc.) • More effective outbreak response x x Outputs x Health system supervisory data collection Analysis and info sharing Decision-making and action to address needs • Records may not be compiled • Few records may be collected • Some records may be lost • May have insufficient analytical tools to do trend and other analyses • Analytical needs at various levels may not be met • May not have clear use of data by decision makers • May not have clear tracking of decisions made with data 5

7. Country example: Senegal yielded a tangible example of impact Case Study: Partogram Usage in Senegal • Context • In Senegal, there is evidence that the process is working: data is collected, analyzed and then used as an input into decision making. • Given that foundation, the team identified a specific instance where the deployment of the PDA and integrated survey can be reasonably linked to an improved health outcome. • Story • Initial PDA collected data showed low usage of the partogram, a simple form used by midwives to monitor the status of a delivery [at left]. • Delivering babies without using the partogram increases the chances of complications. • Seeing this gap in his district, the Pikine District Chief ordered a tripling of production of the forms and ensured they were distributed to midwives. • Between March and August 2008, data collected with the PDA showed an increase in partogram usage in the pilot districts by 14 percentage points, even though usage remained unchanged outside those districts. Just one example • The partogram case study is one example of estimated health impact. • Two-thirds of the 82 questions tracked by EpiSurveyor showed improvement and increased by 10% or more. • Additional case studies may be feasible based on previous data and continued support for data collection in the country.