Christopher J.L. Murray Richard Saltonstall Professor of Public Policy Director Harvard University Initiative for Global

1. Monitoring and Evaluation of Financial Reforms Christopher J.L. Murray Richard Saltonstall Professor of Public Policy Director Harvard University Initiative for Global Health Harvard University Global Health

2. OUTLINE Scope What to Monitor? Data Systems Technical Challenges Accountability and Global Public Goods Harvard University Global Health

3. Scope: Total Health System vs Subsector Health financing reforms can have unintended consequences on different parts of the health system Sub-sector specific changes should be evaluated in terms of changes in the overall health system wherever feasible Vertical vs horizontal program debate Harvard University Global Health

4. OUTLINE Scope What to Monitor? Data Systems Technical Challenges Accountability and Global Public Goods Harvard University Global Health

5. General and Specific All reforms should monitor and evaluate key general aspects of the health system including inputs, the provision function, and outcomes For each reform there are likely to be specific factors that should also be monitored, e.g. affilitation in the Mexican reform Harvard University Global Health

6. What to Monitor: Inputs Total health expenditure Health expenditure flows to sub-sectors and providers Human resources: composition, total quantity and geographic distribution Harvard University Global Health

7. Coverage skilled birth attendance vs Nurses, Midwives and Doctors Harvard University Global Health

8. What to Monitor: Provision Number and location of providers Drugs and medical equipment Technical quality of providers Effective coverage of key interventions Harvard University Global Health

9. Technical Quality Operational Definition Quality of a provider is the proportion of potential health gain for its set of patients that it delivers. Harvard University Global Health

10. What to Monitor: Outcomes Average level of health Health inequalities Responsiveness Distribution of household health payments Harvard University Global Health

11. Short-Term vs Medium Term Changes Reform effects will appear over a shorter period of time for: expenditure, technical quality, effective coverage, responsiveness and distribution of household health payments Longer term: human resources, number and location of providers, health and health inqualities Harvard University Global Health

12. OUTLINE Scope What to Monitor? Data Systems Technical Challenges Accountability and Global Public Goods Harvard University Global Health

13. Data Systems Provider activity registries – hospital discharges, clinic visits, intervention delivery, complications, procedures Provider facility surveys – drugs, equipment, technical quality knowledge Household surveys/panels – population health (level and distribution), effective coverage (not all interventions), responsiveness Harvard University Global Health

14. Data Systems (II) Vital registration – mortality Budgets/Disbursements – distribution by sub-sector of expenditure Record linkage – focus on distributional concerns increases utility of record linkage between provider registries, household surveys and vital registration Harvard University Global Health

15. OUTLINE Scope What to Monitor? Data Systems Technical Challenges Accountability and Global Public Goods Harvard University Global Health

16. Confounding Major issue for evaluation of reforms is potential for confounding e.g. those included in a reform differ in some unmeasured variable and problem of temporal confounding Randomized study design (phase-in) Matching Other strategies to come as close to randomization as possible Harvard University Global Health

17. Drowning in Detail Monitoring and evaluation of health reforms frequently generate data on a large number of indicators Multiple proxy measures often used when quantity of interest is difficult to measure Critical to summarize different domains of monitoring and evaluation with composite measures Harvard University Global Health

18. Provider Registry Bias and Gaming If monitoring or evaluation has financial, or other consequences on providers, strong incentives exist that can lead to biased information from providers. Process of reform may change the incentives and magnitude of bias – gaming Must stay one step ahead of gaming in quality review of data Harvard University Global Health

19. Comparison of DHS estimates and officially reported DPT3 coverage Harvard University Global Health

20. Measurement Error and Distributional Analysis For distributional analysis (household health expenditure, effective coverage, responsiveness and health) increased measurement error will appear as increased inequality Statistical methods required to decompose measured variation into stochastic and systematic components – not yet developed for household health expenditure Harvard University Global Health

21. Technical Quality Metrics Standardization of methods to measure technical quality for hospitals and ambulatory care Issues of risk adjustment and use of mutiple indicators to estimate quality Harvard University Global Health

22. OUTLINE Scope What to Monitor? Data Systems Technical Challenges Accountability and Global Public Goods Harvard University Global Health

23. Building Evidence on Health Systems Cross-national research on health systems can provide insights into what works and what does not Evidence on health system organization requires a common framework, definitions and metrics for inputs, processes, coverage and outcomes Investment in health and health system metrics can have a long-term as well as short-term benefit Harvard University Global Health

24. Fostering A Culture of Accountability A culture of accountability for outputs (coverage) and outcomes maybe the most important ingredient for health system performance Implementation challenges can be solved by social entrepreneurs if accountability mechanisms create the right incentives Culture of accountability requires metrics with three characteristics: credibility, clarity and comparability Harvard University Global Health

25. Metrics for Accountability Credibility -- valid, reliable measurement with an explicit data audit trail Clarity -- metrics that the media, general public and decision-makers can understand; clarity is not necessarily simplicity; requires very limited number of core measures with supporting disaggregation for technical specialists Comparability -- metrics need to be comparable overtime and across communities and nations to broaden the scope and relevance of accountability Harvard University Global Health