The National Evaluation Platform (NEP ): An introduction for Tanzania stakeholders

1. The National Evaluation Platform (NEP): An introduction for Tanzania stakeholders

2. Most current evaluations of large-scale programs aim to use designs like this Coverage No coverage Impact No impact No program Program

3. But reality is much more complex General socioeconomic and other contextual factors Other health programs Routine health services Interventions in other sectors Coverage Impact Program Other health programs Many Stakeholders Gov’t Ministries DPs NGOs

4. This presents important challenges for evaluation and accountability • Simultaneous implementation of multiple programs • Separate, uncoordinated, inefficient evaluations, if any • Inability to compare different programs due to differences in methodological approaches and indicators The NEP aims to help Governments and their partners meet these challenges

5. The NEP approach • Supports answers to “what works” questions in accelerating intervention coverage and reductions in mortality and undernutrition • Addresses integration by assessing multiple programs (i.e.iCCM, nutrition, immunizations, WASH) simultaneously over time • Promotes country ownership of data, builds capacity for evidence-based policies and programs • Facilitates global and national accountability reporting (e.g. MDG, CoIA, SUN, etc)

6. Common Evaluation Framework

7. Evaluation data systemwith districts as the units • District-level databases covering the entire country combining data from multiple sources • Containing standard information on: • Inputs (partners, programs, budget allocations, infrastructure) • Processes/outputs (DHMT plans, ongoing training, supervision, campaigns, community participation, financing schemes such as conditional cash transfers) • Outcomes (availability of commodities, quality of care measures, human resources, coverage) • Impact (mortality, nutritional status) • Contextual factors (demographics, poverty, migration) Permits national-level evaluations of multiple simultaneous programs

8. A single, integrated data base with districts as the rows Core Data Points from Other Sectors Core Data Points from Health and Nutrition Sectors Nutrition Surveillance System Women’s education Climate Factors National Health Accounts Quality Checking & Feedback to Source DHS HMIS

9. Collect new data only as required to fill gaps • Partner mapping • Strength of program implementation • Quality of care • District-level contextual factors

10. Types of NEP Evaluation Questions • Interim (formative) analyses • Are programs being deployed where need is greatest? • Is implementation strong enough to have an impact? • How to best increase coverage? • How can programs be improved? • Summative analyses • Did programs increase coverage? • Was coverage associated with impact? • Did programs have an impact on mortality and nutritional status? • Are there alternative explanations for the findings? • How equitable are the programs? • How much did programs cost?

11. Types of comparisons supported by the NEP approach • Districts with or without a given program • Traditional before-and-after analysis with a comparison group • Dose response analyses • Regression analyses of outcome variables according to dose of implementation • Stepped wedge analyses • In case program is implemented sequentially

12. NEP Project Objectives With support from Canada DFATD: • Develop NEPs in 4 countries, and build local institutional capacity to use and maintain them • Demonstrate that NEPs can provide high-quality, timely data on results and implementation strength for use in guiding decisions • Use the experience to develop guidelines and tools for use in additional countries

13. NEP Project Overview • 39 months starting August 2013; 3 months’ start-up and three years’ full implementation • In each country: • Scope: At least MNCH & nutrition; could broaden if called for in a particular country • Advisory Committee: Need high-level buy-in from Government and major partners; can be existing country group • In-country institutional partner: Credible, stable, opportunities to build sustainable institutional capacity • Project resident advisor: Can be national or international

14. Major Activities • Entry • Conduct situation assessment • Meet with stakeholders • Select in-country home institution(s) • Identify Advisory Committee • Building and populating the data system • Existing data (HMIS, household surveys, service assessments, budgets) • New data as required (partner mapping, implementation strength assessments, quality of care assessments, cost-effectiveness)

15. Major Activities - 2 • Establishing processes • Data quality assessments • Feedback to data source for quality improvement • Analysis and response to Government/stakeholder questions • Continuous improvement of NEP functionality • Using the NEP to strengthen programs • Use of NEP evidence to generate questions and support evidence-based decision making • Improved reporting on key indicators for CoIA, SUN, etc. • Regular feedback to programs on implementation strength, quality of care, etc. • Small grants program in program-relevant gender analyses

16. Capacity Building • Mentoring and apprenticeship for home institution(s) • Technical work for NEP data system development, maintenance and analysis • “Learning by doing” approach, with continuous TA that evolves over time • Formal skills-building workshops with policy makers and program planners • Bringing together NEP home institution and relevant Government counterparts (e.g. ministry planning /M&E units, TWG members, etc) • Focus on formulation of policy-relevant evaluation questions and interpretation/application of NEP findings • Tailored to country-specific needs • Cross-country teaching and learning encouraged

17. Proposed strategies for NEP capacity building Ongoing mentorship and technical assistance to build capacity in NEP home institution (s)

18. Discussion How can the NEP project Best contribute in TanZania?

19. Asanteni SanaThank You

20. What are priority questions for the NEP? • What answers do Government and partners need answers to in order to deliver stronger programs for women and children? e.g., • Do districts with coordinated child survival and nutrition programs achieve better outcomes? • Are districts using CHW strategies achieving better results? • Are RMNCH, Nutrition & HIV/AIDS services truly reaching the poorest and most in need?

21. What should be the scope of the NEP in Tanzania? • Nutrition is an essential element of women’s and children’s health. • How can the Scaling Up Nutrition work being led under PMO-RALG and TFNC be fully integrated into the NEP approach?

22. Source: MOHSW (Sep 2013) MIDTERM ANALYTICAL REVIEW OF PERFORMANCE OF THE HEALTH SECTOR STRATEGIC PLAN III 2009–2015

23. NEP Interventions EQUITY ANALYSIS Disaggregation by gender, wealth, urban/rural, region for core coverage indicators measured through household surveys # TZ HSSP III indicator ǂ CoIAindicator

24. NEP Impact Indicators • Maternal mortality ratio #ǂ • Under-5 mortality rate #ǂ • Neonatal mortality rate • % Stunting #ǂ • % Wasting # • Cost per life saved, by age Proposed nutrition-focused • % anemia women 15-49 y • % anemia in 6-59 month olds • % Acute Malnutrition U5 # TZ HSSP III indicator ǂ CoIAindicator

25. Key Contextual Factors

26. How might the NEP best be organized in Tanzania? • Are there groups that can contribute to developing and sustaining the NEP? Who? • Is there an existing advisory group(s) that would be suitable for the NEP? • If not, what would the recommended composition of an NEP-specific advisory committee?

27. How can the NEP contribute to existing Government efforts to strengthen evidence and accountability for women’s and children’s health? • Examples of current efforts • Tanzania’s leadership in the Commission on Information and Accountability for Women’s and Children’s Health • Ongoing improvement of the HMIS • Scaling Up Nutrition (SUN) monitoring activities • Global initiatives like Countdown to 2015, A Promise Renewed, Open Data Initiative etc.