Three Pillars of M&E

A Comparison of Different Methods of Data CollectionHealth System Innovations Workshop Abuja, Jan. 25-29, 2010

Three Pillars of M&E • Setting Objectives and selecting Indicators √ • Collecting data using appropriate methods • Using the data to make decisions and improve performance

Important Methods Different methods of data collection required, each with benefits & issues • Routine reporting system (HMIS): reported data coming from service providers • Household surveys (HHS): statistical sample of households in the community • Health Facility Assessments (HFA): a survey of a sample of health facilities to assess QOC • Quantitative supervisory checklist

1. Routine Reporting - HMIS Advantages • Usually near real time • Highly Disaggregated • Local managers can their manage own performance Disadvantages • often inaccurate • service providers have incentive to overstate • difficult to provide data on equity, quality • can’t assess effects at community level • Expensive (HW time) 8x cost of HHS • Misses info from private sector

Excessive & Exclusive Reliance on HMIS is Fraught with Danger • Murray et al have found that GAVI ISS may have over-paid by 93% • Data Quality Audit (DQA) was no help in discovering discrepancies between HMIS & surveys • People want to tell their superiors good news • May worsen when tied to some form of bonus

HMIS vs. Household Survey DPT3 Coverage in Pakistan Average difference = 14% points

2. Household Surveys Advantages • More accurate • Can be independent of service providers • Provides data on coverage, equity, expenditure, CPR, ITN use, satisfaction, etc. Disadvantages • Relatively expensive (cheaper than HMIS but an incremental cost) • Episodic, not done in real time • Level of disaggregation

M&E: More Frequent Household Surveys • In a rapidly changing system need more frequent data • Data required to verify HMIS • Lots of data best obtained from HHS • Household behaviors (use of ITNs, CPR, etc.) • Utilization, patient satisfaction, payments • Knowledge and behaviors • Source of care (private, public, non-qualified) • Suggestion: Every 2 years with disaggregation (LQAS or cluster surveys at state level)

3. Health Facility Assessments Advantages • Can assess quality of care, more than anecdotes • Can be independent of service providers • Can be done more often than HHS • Relatively inexpensive Disadvantages • Complex to design • Lots of data, can overwhelm managers • Cannot provide information on coverage, equity

4. Quantitative Supervisory Checklist • A reduced version of a health facility assessment • Objectively assesses a variety of indicators to come up with total score. • Takes about 1-3 hours to complete • A copy of results left in the health facility, easy to track progress • QSC is both a management intervention and tool for M&E

4. Quantified Supervisory Checklists Advantages • Can assess QOC. • Can be independent of service providers • Can be done often • Inexpensive • Clarifies what is expected of HWs • Can be adapted to conditions as they change Disadvantages • Challenging to design • Cannot provide information on coverage, equity • Ensuring continued use is difficult

Recommendations • Use all four approaches – none are perfect • Relying excessively on HMIS is unwise • Allocate sufficient budget to carry out various approaches to data collection (5-7% of projects) • Make M&E somebody’s job

Three Pillars of M&E