Monitoring health system performance - s ynthesis of some experiences from low-income countries

1. Monitoring health system performance - synthesis of some experiences from low-income countries Dina Balabanova, Tim Powell-Jackson, Richard Coker, Kara Hanson & Anne Mills London School of Hygiene and Tropical Medicine Health System Metrics, Glion sur Montreux, 28-29 September 2006

2. Overview • Background • Complexity • Objectives and methods • Measurement • Health financing • Health care delivery • Emerging issues • Conclusions

3. Background • Commitment to invest in health systems is unprecedented, but will not last unless it is possible to show results • Currently poor health information available but demand for improved health system metrics (national / international) • Opportunities • Health System Metrics and other initiative seeking to strengthen HIS • Commitment to the health MDGs – need to measure progress • Growing consensus of importance of measurement strategies & monitoring & evaluation built into programme planning cycles • Threats • Limited resources for health information and sustainability • Capacity constraints (in the health and social sectors)

4. Objectives & Methods • Purpose of the study: a review of some low-income countries’ experiences with health system performance monitoring and use of data • Case study countries: • Georgia • Rwanda • Uganda • West Bengal, India • Material from other countries • Selection criteria • Analytical approaches: • uses the WHO health system performance framework • synthesis around common themes and issues • identifying unique lessons in each type of context

5. Complexity • How should health system performance be measured? • Increasingly multiple contacts with the system, chronic diseases • Outcomes determined by different care components, sectors • Need for system-wide and inter-sectoral indicators • Tension between international (donor-driven) demands and country-level agendas and needs • Use of normative approaches imply causality • To what extent monitoring influences policy? • Impact of measurement on health systems, e.g. Indicators that are measured often improve • Monitoring information may be complex to interpret where a range of interventions co-exist.

6. Measurement What approaches are taken to measure health system performance in the study countries?

7. What is measured ?

8. Health financing How has information been used? Where are the gaps? What challenges remain?

9. Use of health financing information • Identification of financing gaps and advocacy for increased allocation of funds to health (Rwanda) • Health sector leadership and management of funds (Tanzania, Rwanda) • Equity of health financing in the health system (South Africa, Rwanda) • Protection against the financial burden of ill health (Mexico) • Resource allocation with the health sector (Rwanda)

10. Gaps in health financing information • Private health expenditures – difficult to collect compared to public and external health financing sources • Coverage of NHA relatively low in developing countries but expanding • Health financing data at decentralised levels for local decision-making • Financial burden of ill health and impact on impoverishment at the household level

11. National Health Accounts in Africa Number of NHA Rounds 1994 – 2004

12. Remaining challenges • Institutionalisation of NHA into the routine activities of Government • Underlying problems in Public Expenditure Management systems and data reliability • Timeliness of data (NHAs and household surveys) • Collection of private expenditure health financing data • Tension between disease expenditure and general health expenditure financial tracking • Addressing the needs of in-country policy makers vis-à-vis that of external agencies

13. Health care delivery How has information been used? Where are the gaps? What challenges remain?

14. Use of information: country examples West Bengal, India Aim: to monitor the performance of public sector programmes. Improve accountability and planning at national level • Standard service use indicators & regular meetings in PHC facilities Uganda Aim: to link health system performance monitoring to SWAPs and national policy process. Allows policy adjustment. • Data used in the annual health sector review process and to inform the development of annual plans • District league tables to rank performance of districts & motivate districts to improve indicators. • Tracking surveys – at the start of SWAP, 2001- to assess Govt systems (financial procedures, drug distribution, HR deployment)

15. Major gaps in measurement • Private sector – service use, service availability (infrastructure, human resources, services offered) • Vital events • Efficiency of health system • Quality of health care • Effective coverage

16. Remaining challenges • Low capacity and motivation to use data: • Locally • For decision-making or for policy initiatives • Lack of ownership by health providers, who are not involved in designing of monitoring procedure and indicators • Capacity for analysis concentrated at central level • Feedback to lower levels is limited, poor internal feedback • HMIS is often mistrusted • Selection of indicators often creates distortions • Information systems do not reflect move from project to system performance • India: ‘critical milestones’ & vertical project indicators

17. Emerging issues

18. Data quality and reliability • Existing information systems, but data inaccessible or inappropriate to needs and policy process • Developing parallel monitoring frameworks rather than adapting & use of existing data: concerns for complexity and data reliability • HIS not always reflecting reform developments • Limited external data audit and reliance on single data sources (Rwanda, Uganda) • Technology involved in data collection, analysis and use often rely on bespoke software.

19. Parallel systems • Donor agenda regarding data collection, unsustainable • Data collection, analysis and use for policy is fragmented • Uganda/Nepal: lack of unified data linked to SWAPs • Private sector is often not covered (India/Uganda) • Multiple reporting requirements (Rwanda/India). • Lack of inter-sectoral information systems and unified quality standards. (Uganda/ Rwanda) • Vertical donors-supported programmes often function well in the short-term but may distort wider systems (e.g. Georgia & Angola)

20. Information flows & level of use • One-way traffic for information • Disaggregated data not available at sub-national level • Information intended to be used locally, is used at national level, or for different purpose reflecting governance & aid coordination • Information that is not aggregated nationally, less useful internationally • Governance and stewardship at local level needs to be able to draw effectively on aggregate & disaggregate data • Disaggregated data feeds effectively into local planning when linked to decentralised decision-making (TEHIP) • Peer comparisons at district level – productive vs unhelpful

21. Factors facilitating measurement & use of data • Health system monitoring embedded within reform process • SWAPs/ PRSP in Uganda, Rwanda; district autonomy (TEHIP) • Unintended consequences (Afghanistan) • Selective use of data internationally (user fees/HIV, in Uganda) • In post-conflict settings, the aid influx promotes monitoring health systems & early warning systems. Possible inefficiencies. • The importance of governance • Channels for policy exist (annual reviews, SWAPs meetings) & comparable timelines. • Communities and non-health system stakeholders involved • Large-scale data collection exercises are resource-intensive and not synchronised with the policy process (some In-DEPTH/ LSMS). • Technology, appropriate to context

22. Conclusions Effective health systems monitoring requires: • Capacity: to collect or use existing data, analyse, inform policy • Ownership • Coherence between domestic and external demands • Coherence between external agencies • Coherence between system-wide monitoring and vertical programmes performance measurement • Coherence between assessing the performance of different system elements • Domestic governance • Impact measurement to ensure sustainability/reform (scaling up) • Foster partnership between stakeholders

23. Acknowledgements Georgia George Gotsadze India Barun Kanjilal Rwanda Vianney Nizeyimana Tanzania Graham Reid Uganda Valeria Oliveira-Cruz Freddy Ssengoba