Metrics, Targets & Performance: The Case of NHS Trusts

1. Metrics, Targets & Performance:The Case of NHS Trusts Philip Stevens, NIESR Mary O’Mahony, NIESR & University of Birmingham

2. Introduction • The aim of performance measurement and target-setting by government: • To improve the quality, performance and accountability of public services • But do targets achieve this aim? • Comprehensive performance measures are required, which cover the full range of services being delivered, to evaluate: • Extent to which targets are met • Unforeseen adverse consequences of target-setting • Otherwise, providers may concentrate on areas that are measured to the exclusion of others

3. Aims & Objectives of the Research • Examine the desirable properties of performance measures • Consider performance measures that might be applicable to NHS Trusts • Compare alternative methods of assessing performance • Consider the impact of targets on behaviour

4. Star Rating System • Introduced in 2000/1 – Final year 2004/5 • Will be replaced by ‘annual health check’ in 2005/6 • Small set of Key Targets • A larger set of ‘balanced scorecard’ indicators

5. Star rating system

6. Key targets used in star ratings for acute trusts

7. Alternative performance measures • Performance indicators • Frontier analysis • Productivity analysis

8. Performance indicators Complex composite indicators. Methodological issues: Nature of the underlying data Transformation of the data Weighting system Alternative performance measures

9. Frontier analysis Two main methodologies: Data envelopment analysis Both have strengths and weaknesses. Generally, they are subject to many of the same problems as productivity indices E.g. measurement of outputs Alternative performance measures

10. Alternative performance measures • Productivity analysis • What you get out for what you put in • The ratio of the volume of outputs to the volume of inputs • What Productivity isn’t • Efficiency

11. Public Sector Productivity • Many problems in assessing public sector productivity • Difficulty in identifying the outputs • Lack of market prices • Attribution • Outputs previously measured by inputs • Makes productivity measurement impossible, by definition

12. The price of everything, but the value of nothing • Why would we want prices? • Quality change • Adding apples and oranges • Products and services have a number of characteristics that are valued by consumers • For example, when we compare computers, we compare CPU speed, memory, hard-discs etc. • Hedonic regressions of prices on characteristics

13. Applying the private sector method to public services • Lack of information on prices • Need measure of marginal benefit to the consumer • Cost shares have often been used as weights but may be divergence between marginal costs and benefits • Could use value judgement weights but this is controversial • Could attempt to impute values

14. Measuring NHS outputs and productivity • Important to distinguish between • Activities (e.g. operative procedures, diagnostic tests, outpatient visits) • Outputs (e.g. courses of treatment - may require a bundle of activities) • Outcomes (e.g. the characteristics of output of value to individuals) • In private sector focus is on outputs • Prices value characteristics • Activities embedded in outputs • No markets and hence prices for NHS outputs

15. Obtaining marginal valuations • Two ways of valuing output: • Measure the outputs and attempt to estimate the marginal valuations attached to them • Measure the outcomes produced by each unit of output and attempt to estimate marginal valuations of outcomes • Allowance must be made for output quality: • Greater health gains • Reduced waiting times

16. Implementation • Cost-weighted activity index (CWAI) • Implies costs reflect valuations at the margin • Does this hold for the NHS? • May be no practical alternative in the short run • Valuation of outcomes • Health outcomes • Readmission • Waiting times • Choice and certainty of date of treatment • Patient satisfaction • Environment • Attribution

17. The empirical analysis • Analysis restricted to Acute trusts • Output data come from Hospital Episode Statistics • Finished consultant episodes vs. continuous inpatient spells • Input data from • NHS employment census • Trust financial returns • Earnings surveys (NHS and national surveys)

18. Next Steps • Results of comparison of star ratings with alternative indicators to be presented at conference before the end of the year • Results of analysis of micro effects of choice of indicators to be presented at seminar in the new year • Special issue of the National Institute Economic Review on the analysis of public sector performance in July

19. Public sector conference • Following on from previous conferences at the British Academy in 2002 and 2003 • Focus on dissemination and user engagement • Post-conference website to bring together academic work itself and responses, in particular from those at the coalface • Followed up by special issue of National Institute Economic Review