Monitoring and Evaluation: Calculating and Interpreting Coverage Indicators

1. Monitoring and Evaluation: Calculating and Interpreting Coverage Indicators

2. Learning Objectives By the end of the session, participants will be able to: • Identify sources of data for calculating coverage indicators • Estimate denominators for routine coverage estimates • Calculate and interpret coverage indicators from routine data • Use online resources for estimating coverage indicators • Assess the quality of relevant data sources • Reconcile coverage estimates from different data sources

3. Maternal Health Coverage Indicators • Proportion of pregnant women who received at least two antenatal care visits • Proportion of deliveries occurring in a health facility • Proportion of deliveries with skilled attendant at birth • Proportion of women attended at least once during postpartum period (42 days after delivery) by skilled health personnel for reasons related to childbirth

4. Why Coverage Indicators Are Important • Understand how effective program is • See if one target group is reached more effectively than another • Identify underserved area/regions

5. Child Health Coverage Indicators • Immunization Programs • DTP3 vaccine coverage • Measles vaccine coverage • BCG vaccine coverage • OPV3 coverage • HepB3 coverage • Fully immunized child • Nutrition programs? • Control of diarrheal disease programs?

6. Coverage Indicators for HIV/AIDS Care & Treatment Programs • Number of clients receiving public/NGO VCT services • Number of clients provided with ARVs • Percent of children in need receiving cotrimoxazole prophylaxis • Percent of HIV patients receiving DOTS • Coverage of PMTCT programs?

7. Where Do We Get the Data? • Censuses • Surveys • Registrations • Health management information systems • Program statistics • Patient registers

8. Estimating Coverage From Routine Data

9. Indicators From Program Statistics: Numerators • HMIS and routine reports give information on numerators • Numerators: number of deliveries in health facilities, measles vaccinations, pills distributed, voluntary counseling and testing clients etc. • Denominators: ?

10. Town A vaccinated 200 infants Town B vaccinated 400 infants Town C vaccinated 600 infants Population size: Town A= 10,000 Town B= 30,000 Town C= 60,000 Example:Importance of denominator

11. Indicators From Program Statistics: What Denominators Are Needed? • Denominators: population composition • Population composition • How many women are of childbearing ages? • How many children are under five? • How many adolescents? 15-19? 20-24? • How many men are 15-59 years? • How many children are of school going age? • How many infants are there? • How many babies are born each year?

12. How Do We Get Denominators? • Population registers • Censuses • Population projections • Population growth rate (r) • Rate of natural increase = crude birth rate (CBR) minus the crude death rate (CDR) • Net migration rate: inmigration - outmigrants per 1000 population • CBR: no. of births per 1000 population in 1 year • CDR: no. of deaths per 1000 population in 1 yr • Population growth = rate of natural increase + net migration rate

13. Spectrum Model • DemProj: projects population of country/region by age and sex based on assumptions about fertility, mortality, and migration • Urban and rural population projections can also be prepared • EasyProj: supplies data needed to make a population projection from estimates provided by the Population Division of the UN www.tfgi.com

14. Spectrum

15. Calculating Denominators • Population at time t: P(t) = P(0) * exp(r*t), where: • P(t) is the population size after t years • P(0) is the population size at the last census • Example: • 300,000 people at census • Growth rate = 3% (0.03), • What is the population after 10 years? • 404,958 people

16. Estimating Number of Live Births • Where data on the number of live births are unavailable: Total expected births = Total population x crude birth rate • Where the crude birth rate (CBR) is unknown: Total expected births = Total population x 0.035 Source: WHO 1999a; WHO 1999b

17. Estimating Number of Surviving Infants • Target population for childhood immunization: Surviving infants <12 months of age in a year • Where data on the number of surviving infants are unavailable: Total expected number of surviving infants = Total population x CBR x (1 – infant mortality rate)

18. Estimating Number of Surviving Infants: CBR Known Total population: 5,500,000 CBR: 30/1000 Infant mortality rate (IMR): 80/1000 Number of surviving infants = Total population x CBR x (1 – IMR) = 5,500,000 x 30/1000 x (1 - 0.080) = 5,500,000 x 0.030 x 0.920 = 151,800 Source: Immunization Essentials: A Practical Field Guide (USAID, 2003)

19. Estimating Number of Surviving Infants: CBR Unknown • Where data on the number of surviving infants, CBR or IMR are unavailable, multiply total population by 4%: Expected no. of surviving children < 12 months = Total population x .04 • If the total population is 30,000, then the number of children under one year = 30,000 x 4/100 = 1200 Source: WHO, 2002b

20. Estimating the Monthly Target Population Monitoring immunization and vitamin A coverage should be done monthly at the facility and district levels, requiring estimations of the monthly target population Monthly target population = Estimated number of children under 1 year of age divided by 12 Example: • Annual target population of children < 12 months = 1200 • Monthly target = 1200/12 = 100

21. Example: Immunization Coverage From Routine Data • Total population of district in 1990 = 99,000 • CBR = 40 per thousand • IMR = 80 per thousand • Population growth (r) = 3% per year • 3,000 measles vaccinations were given to infants in district in 1998 • What is the measles coverage rate for 1998? • Numerator: No. immunized by 12 months in a given year • Denominator: Total no. of surviving infants < 12 months in same year

22. Immunization Coverage From Routine Data: Answer • Estimate district total population in 1998 Pop1998 = 99,000 * exp(.03*8) = 125,854 • Estimate number of surviving infants in 1998 125,854 x (40/1000) x (1 - .080) = 4615 • Estimate measles coverage rate Measles coverage = 3000/4615 x 100 = 65%

23. Challenges in Estimating Coverage from Routine Data • Limited knowledge of target pop/denominators • Low timeliness & completeness of reporting • Poor data quality • Lack of written standard reporting procedures • No systematic supervision on data management • Dual reporting systems (EPI, HMIS) • Inclusion of data from private sector

24. Assessing Reliability of Routine Coverage Indicators • Understand how denominators are derived • Understand the process of collecting the information • Look for inconsistencies and surprises

25. Assessing Reliability of Routine Coverage Indicators • Look for reliable data from other sources to use as a basis for comparison • Cross-check

26. Estimating Coverage from Survey Data

27. Survey Tools for Coverage Estimation • WHO-EPI surveys • Lot quality coverage surveys • Large-scale population-based surveys • USAID Demographic and Health Surveys • UNICEF Multiple Indicator Cluster Survey • Arab League PAPCHILD surveys • CDC Reproductive Health Surveys • Seventy-five household survey • Knowledge-Practice-Coverage Surveys • Other local surveys

28. How Do Administrative Data Compare With Survey Data?

29. Reconciling Coverage Estimates From Different Data Sources • Age group & geographic scope • Health cards versus recall • Different sources for different purposes • Not all coverage data can be compared in constructive way • Differences in inclusion of private sector • Selectivity

30. On-line Resource: STATcompiler • Innovative online database tool • Allows users to select numerous countries and hundreds of indicators to create customized tables that serve specific needs • Accesses nearly all population and health indicators published in DHS final reports http://www.measuredhs.com/statcompiler

31. STATcompiler

32. On-line Resource: DOLPHN • DOLPHN: Data Online for Population, Health and Nutrition • Online statistical data resource • Quick access to frequently used indicators from multiple sources, including: • DHS, BUCEN, CDC, UNAIDS, UNESCO, UNICEF, World Bank, WHO www.phnip.com/dolphn

33. Advantages and Disadvantages of Routine-based Coverage Advantages • Provides information on more timely basis • Makes use of data routinely collected • Can be used to detect and correct problems in service delivery Disadvantages • Denominator errors • Poor quality reporting

34. Advantages and Disadvantages of Survey-based Coverage Advantages • Avoids problems with denominators • Includes information from non-reporting facilities Disadvantages • Coverage survey has low precision • Larger standard errors at sub-national levels • Irregular and expensive • Survey timing may affect coverage rates

35. References • WHO. 1999a. Indicators to Monitor Maternal Health Goals: Report of a Technical Working Group, Geneva, 8-12 November 1993. Division of Family Health Geneva: WHO. • WHO. 1999b. Reduction of Maternal Mortality: A Joint WHO, UNFPA, UNICEF, World Bank Statement. Geneva: WHO. • WHO (2002) Increasing Immunization at the Health Facility Level. Geneva, Switzerland: World Health Organization

36. Case Study 1: Immunization Coverage from Facility Data • Estimate total population in 2003 • Calculate coverage for DTP1, DPT3, and measles vaccine in 2003 • Evaluate trends in coverage • Estimate drop-out rates • Analyze the problems in 2003 • Is coverage low or falling? • What are possible causes? • What are the differences in coverage in different areas? • What action can managers take if coverage data indicate problems?