Monitoring and Evaluation: Information Sources and Systems

1. Monitoring and Evaluation: Information Sources and Systems

2. Session Objectives At the end of this session, participants will be able to: • Name the main information sources for PHN M&E • Describe the main strengths and weaknesses of different data sources • Discuss the main data-quality issues that need to be considered • Explain why complementary data sources are often required to monitor and evaluate health systems • Identify potential data sources that might apply in a specific program context.

3. Overview • Types of information • Strengths and weaknesses of selected data sources • Data quality • Linking data sources • Exercise

4. The Finagle’s Laws of Information The information you have is not the information you want The information you want is not the information you need And the information you need is usually not available

5. Definitions (1) • Data: the raw facts that are collected and form the basis for what we know. • Information: the product of transforming the data by adding order, context, and purpose • Knowledge: the product of adding meaning to information by making connections and comparisons and by exploring causes and consequences

6. Definitions (2) • Health system “all resources, organizations and actors that are involved in the regulation, financing, and provision of actions whose primary intent is to protect, promote or improve health.” (WHO, 2000) • Program A set of procedures to conduct activities. The objective is normally the solution to a problem Neither a health system or program is a static phenomena. They experience a continuous process of changes due to pressure from both outside the system and from within the system.

7. Definitions (3) • Health Information System (HIS): A health-information system (HIS), similar to a health management information system (HMIS) “…a system that provides specific information support to the decision-making process at each level of an organization” (Hurtubise, 1984) • Data Systems “a way of talking about the whole set of M&E indicators in a performance monitoring-and-evaluation plan, and all of the data and other information that need to be gathered and understood in an orderly fashion that makes sense and help in program management and implementation”

8. Types of Information • Surveillance • Epidemiological • Behavioral • Routine service reporting • Special program reporting systems • Administrative systems • Vital registration systems • Facility surveys • Household surveys • Censuses • Research and special studies

9. Frequency of Data Collection • ROUTINE or continuous data collection • NON-ROUTINE or periodic data collection

10. Routine Non-Routine Classify the previous information types by frequency of data collection

11. Frequency of Data Collection • ROUTINE or continuous data collection • Health facility-based (patient information and service statistics) • Community-based (service-statistics) • Program-based (administrative) • Vital registration • Sentinel reporting/demographic surveillance • NON-ROUTINE or periodic data collection • Household or facility-based surveys • Population census • Rapid-assessment procedures (RAP) • Special studies/research

12. Geographic System Levels • National • Sub-national (e.g. district) • Program area

13. The Health Information System: Data for Planning, Monitoring and Evaluation in the PHN Sector Routine Non-Routine TYPE USE Aggregated Service Statistics Aggregated Mgmt Data Aggregated Surveillance Data Financial Data Vital Registration Systems Policy-Making Strategic Planning Program Tracking Disease Surveillance Technical & Logistical Support National Level Population-based surveys e.g. DHS Facility-based surveys e.g. Situation Analysis, SPA Rapid Assessment Methods Aggregated Service Statistics Aggregated Mgmt Data Sentinel Sites Observation Checklist Self-Evaluation (e.g. COPE) Planning (Access) Management (Quality/Efficiency) Supervision (Performance) Disease Surveillance District Level Special Studies e.g. EPI cluster surveys, KAP studies, etc. Facility/ Client Client Records Financial Records Supply Records Facility logbooks/data records Aggregated Community Data Client Mgmt and Follow-Up Health Unit Management Work Planning/Priority Setting Census Birth and Death Records School Records CBD logbooks Drug Revolving Fund records Client Mgmt and Follow-up Supplies Management Community Awareness Community

14. Data-Collection Levels • Policy or program • Service environment • Client • Population • Spatial/geographic

15. Data Sources at the Policy/Program Level • Official documents (legislative, administrative) • National budgets or other accounts data • Policy inquiries • Reputational rankings • Program effort scores

16. Trends in Family-Planning Effort Score: 1972-1999

17. Data Sources: Service Environment Level • Administrative records • Service statistics • Management information • Financial data • Service-delivery point • Routine service statistics • Audits/inventories • Facility surveys • Agent, staff or provider • Performance, competency • Training records

18. EXAMPLE: SERVICE ENVIRONMENT LEVEL DATA

19. HEALTH MANAGEMENT INFORMATION SYSTEMS (HMIS) • Note: An important way of monitoring routine data over time is through a Health Management Information System. An HMIS is a system for ongoing (routine) collection and reporting of data about client-service delivery. In many countries, this system operates at the national level. Ideally, these routine data are collected from a comprehensive set of service delivery points, and should cover topics such as: – Costs – Stockouts – Births – Mortality – Morbidity – Numbers of clients seen, referred (inpatient; outpatient) – Numbers of clients by types of service

20. Data Sources: Client • Client-exit interviews • Case surveillance • Epidemiology of disease • Provider-client observation • Management of the sick child • Vendor-client interaction • Contact or visit registers • Customer record

21. Data Sources: Population • Census • Vital registration system • Sample household surveys • Special population surveys • Demographic (elderly, youth) • Risk groups (CSWs, MSMs, IDUs) • Occupational (farmer, skilled labor) • Area-based (catastrophe-affected) • Biomarkers

22. Spatial/Geographic Data Sources • Satellite imagery • Aerial photography • Digital line graphs • Digital elevation models • Cadastral maps • Global Positioning System data • PLACE (site-based surveys)

23. Aerial Photography Digital Elevation Models Integrated GIS Database Satellite Imagery Cadastral Data Digital Line Graphs GPS Data Chris Betz 1757 Millbrook Ln 28226 Y 2 Christian Carl 1761 Millbrook Ln 28226 Y 1 Chris McAfee 1765 Millbrook Ln 28226 Y 2 Dale Legere 1776 Millbrook Ln 28226 N 6 Donna Black 1780 Millbrook Ln 28226 Y 2 Demographic Data Differential GPS

24. High Transmission of HIV Guguletu, Cape Town South Africa Carolina Population Center University of North Carolina at Chapel Hill Neighborhood Statistics Showing High Transmission Sites within 500 Meters 1 - 13 14 - 25 Air Photo Showing Potential High Transmission Sites in Guguletu 26 - 37 Individual Structures Can be Identified 38 - 49 50 - 61 No Data

25. Strengths and Weaknesses of Selected Data Sources

26. Focus M&E Data Sources • Facility-based routine information systems • Facility surveys • Population-based surveys • Program records/administrative data

27. Facility-Based RHIS: Types of Information Generated • Service statistics • Outcomes of health interventions if individual patient records kept • Not coverage (but can be estimated in some cases with other data) • Not incidence (except nosocomial infections) • Not prevalence

28. What is Wrong with Existing RHIS? • Irrelevance of information gathered • Poor data quality • Duplication and waste among parallel health information systems • Lack of timely reporting and feedback • Poor use of information • Centralization of information management without feedback to lower levels

29. Strengths of Routine Health Information Systems • Continuously collected – suitable for frequent reporting • Existing system – no new data collection; builds local capacity; sustainability • Typically available at lowest administrative levels (e.g. district, facility) • Integral part of health system – direct link to health system actions

30. Common Problems With Facility-Based RHIS • Variation in quality and completeness of reporting • Timeliness of reporting • Difficulty of providing coverage estimates • Indicators may not be exactly what you want in a particular context • May only cover government facilities • Double-counting

31. Facility Surveys: Types of Information • Readiness to provide services (inventory) • Infrastructure, staffing, hours of operation, fees • Health worker knowledge • Provider interviews • Quality of Care • Client-provider observation • Client satisfaction • Exit interviews

32. Strengths of Facility Surveys • Can cover both public and private health facilities • More detailed information than is typically available in routine systems • Can be tailored to specific program needs • Timing can coincide with program implementation • Can combine with population survey for outcome monitoring and impact evaluation • Quality control may be easier than in routine systems

33. Limitations of Facility Surveys • Survey sampling design and analysis may be complex • Expensive, time-consuming • Stand-alone – sustainability concerns; less connected to ongoing program decision-making • Information rapidly outdated, unless repeated – not available regularly • Coverage/sample size constraints • National vs. sub-national • By type of facility • Small client sample sizes for some services (e.g. FP, STIs)

34. Facility Survey Initiatives and Tools • Service Provision Assessment (SPA/HSPA) – DHS • Service Availability Mapping (SAM) – WHO • Quick Investigation of Quality (QIQ) – M/Eval (FP only) • Situation Analysis (SA) – Population Council (FP only) • JICA facility surveys and mapping

35. Population-based surveys: Types of information collected • Knowledge and attitudes • Practices • Coverage

36. Strengths of population-based surveys • Representative of the general population – no selection bias • Wide range of outcome-level indicators can be collected • Program coverage • Well-tested instruments; quality control

37. Limitations of population-based surveys • Coverage; national versus sub-national – not suitable for district-level estimates • Frequency; typically only conducted every 3-5 years. • Cannot detect small changes or changes over short periods of time without large sample sizes (expensive) • Not suitable for some types of information (e.g. retrospective attitudes – recall bias)

38. Household survey programs (national) • Demographic and Health Surveys (DHS) • CDC Reproductive Health Surveys (RHS) • UNICEF Multiple Indicator Cluster Surveys (MICS) • PVO Knowledge Practices and Coverage Survey (KPC) (not national) • CDC Young Adult Reproductive Health Surveys (YARHS)

39. Class Activity (1) How to improve facility-based routine information systems in developing countries?

40. Class Activity (2) What are the determinants of health information systems performance?

41. Data Quality

42. Overview • Types of information • Strengths and weaknesses of selected data sources • Data quality • Linking data sources • Exercise

43. Data Quality Issues • Coverage • Completeness (census, sample) • Accuracy – measurement error; biases; double-counting • Frequency of collection • Reporting flow • Reporting schedule • Public accessibility • Supervision

44. Hierholzer (Am. J. Med 1991; 91; 21-26) has called data the Researcher’s (M&E expert) sand. A lens maker takes sand, refines it, melts it, and through a long process of grinding and smoothing, fashions a lens with which to see the world more clearly. Similarly, a M&E expert takes data, refines it and smoothes it until a clearer picture of nature is revealed. If the sand is dirty or impure, the lens will be cloudy and distorted. If data is unreliable or invalid, the M&E expert’s understanding of nature will be clouded and distorted.

45. By paying close attention to the data collection process from the conception of the data collection document through the editing of the data after it is collected, the M&E expert help keep his “sand” pure so that, in the end, nature may be viewed with much clarity and possible • No amount of sophisticated analysis can salvage either a poorly designed or a badly carried out study

46. Linking data

47. Linking data • Data can be linked from different sources, across different levels, or over time • Linking data appropriately requires planning, preferably prior to data collection • Understanding linked data can provide depth and continuity to enrich otherwise discrete points of information

48. Linking Data Why link? • Survey data sets (e.g., household and facility information) can be linked to compare services available and health outcomes across geographical units • Geographical and survey data can be linked to examine the effects of physical attributes on service utilization • Time series and panel data can help build causal explanations of program or project effects Why not link? • May not be necessary for a given program in a given context • Improper methodology can confuse issues more than explain them • Analyzing linked data more appropriate for • evaluation than monitoring

49. Linking Data Examples • Population and facility data can be linked to ascertain health outcomes correlated with service availability, training, or quality of care (e.g. % of live births in catchment area attended by a trained personnel or % of women exclusively breastfeeding until 6 months among women going to facilities where provider training took place.) • Facility and client data can be linked to learn about program expenditures per new family planning acceptor • Facility and staff data can be combined to provide information about the proportion of clients per provider or the proportion of doctors per facility

50. Population-based data highlight limitations in HIS