Alastair Fischer: Centre for Public Health, NICE Modelling in UK Health Policy Decision Making.

1. Alastair Fischer: Centre for Public Health, NICE Modelling in UK Health Policy Decision Making. Newton Institute, Cambridge, June 2014

2. Running Order • Objective function • Before NICE • After NICE • Social value judgements • The 4 pillars of the evaluation process • About Public Health • How Public Health is not Technology Appraisal • A decision theory approach versus a hypothesis-testing approach • Use of ‘what-if?’ or threshold analysis • New issues: cost effectiveness and infectious disease modelling

3. Objective Function • If the objective is to maximise the health gain from interventions, there needs to be a generic expression for health • The QALY: quality-adjusted life year. 1 QALY= 1 year at full health • Pragmatic • Relatively simple • It is measured using a simple questionnaire. The UK uses the simplest – the EQ-5D • 5 dimensions (reasonably orthogonal), 3 states for each dimension • ‘Utility’ (measuring quality of life) from EQ-5D is from 1(full health) to 0 (dead) but it is possible to go below 0 • A QALY is the time integral of utility. • Gain QALYs by either living longer or at a higher QoL.

4. Problems with QALYs • You name them, QALYs have them • But as a quick and dirty solution to being a generic measure, they work well enough • And in a lot of cases, the same decision would be made whether QALYs or Life Years Gained (i.e. ignoring quality of life) were measured. • From a purist (academic) viewpoint, QALYs fail, but from a pragmatic decision-making viewpoint, they are better for the decision-maker than not having them (which is the realistic alternative).

5. Back to the Objective function • Maximise the QALY gain from a given budget • Classic optimisation problem • All the classic theorems apply (MUX/pX = MUY/pY) • This allows us to be completely efficient • Er…what about equity (= fairness)? • A: Well, we are prepared to trade off some efficiency to be fairer. • Q: So we really should be maximising the gain in a social welfare function that consists of efficiency and equity arguments? • A: Yes • Q: But equity is subjective, isn’t it? So how do we formulate a SWF? • A: We don’t know if this will do more harm than good, because fairness depends on context. So we use the GOBSAT principle. • Q: I’ve never heard of that. What is it? • A: (sheepish) …Good Old Boys Sat Around A Table. • And we may have multiple objectives, not just equity and efficiency. Again, done by deliberation.

6. The threshold • So for efficiency, we buy the cheapest QALYs first. • And keep on buying the next cheapest, and the next, till we run out of money. That maximises the number of QALYs we can gain. • Where we run out of money is called the threshold. • But NICE only controls a relatively small part of the budget. • Other parts of the NHS use time-honoured deliberation, just like most decisions made in other parts of the Civil Service, in every country of the world. • So the threshold was determined essentially by trial and error in the first instance. • Originally £30,000 per QALY, and then more recently in the range £20,000 to £30,000 per QALY. • More recently, a more objective measure has been found that is good in theory but has many problems. It estimates the threshold to be about £15,000 per QALY.

7. Value based pricing (VBP) • A £15,000 per QALY threshold would mean few new drugs would be approved at their current prices. The drug companies would be affronted. The threshold is based on what the NHS could afford to pay. • So another mechanism to pay higher prices for drugs than the threshold would allow has been sought. The DH has looked at VBP, which would include other benefits not connected to the NHS, such as whether a recipient of a drug could go back to work, and if so, go back sooner. The value of the additional production would be added into the ‘threshold’ for that drug. • NICE has questioned this approach on the grounds that it would discriminate against the old, the young, and the unemployed and low paid workers. So equity has been invoked.

8. Before NICE • There was no single place where doctors could get advice on the best ways of treating patients, including getting unbiased advice on drugs and medical devices such as pacemakers. • Some PCTs would allow a particular drug to be paid for by the NHS but others would not. Postcode prescribing was rife. Drug and device companies went from hospital to hospital, GP practice to GP practice, hawking their wares. Information was inefficiently broadcast. • Clinical guidelines existed, but were relatively haphazardly produced. They had no set timetable for completion, somewhat scatty searches for the best evidence, often unrepresentative committees to determine the decisions, a lack of transparency about how a decision was made and patchy consultation.

9. After NICE • All that changed with the advent of NICE. • Draconian timelines: • we know when a decision will be made, sometimes years in advance. • Transparent process. • Many meetings are held in public. • Strict rules of engagement (methods manuals) • Representative committees • Independent advice • Extensive consultation • Citizen’s Council to advise on Social Value Judgements

10. The four pillars • Efficiency • Social Value Judgements • Process (transparency, timeliness, etc) • Policy Perspective (What will change a decision?) • SVJs sometimes override efficiency • The right to vote is inalienable. It cannot be traded. • Additionally, other SVJs can be traded • Equity offsets efficiency to some degree. • Modellers need to be aware of all these things as context to their modelling.

11. About Public Health • Very many interventions in public health are cost effective • Models can often be very simple and do not require fancy extensions to be able to determine cost effectiveness. • PH committees don’t like it when the NICE PH health economist says that an intervention will be in the right direction and will be cost saving. Therefore by definition cost effective!! • The committees seem to feel cheated, and hold out for numbers. Until they get numbers, they appear to fret and to worry about cost effectiveness. • But put a number in front of them – however great the uncertainty – if it comes in far enough below £20,000 (such as £15,000) they take cost effectiveness for granted and move onto more interesting discussions. • A caricature? Decidedly so, but as with all good caricatures, the exaggeration is based on enough grains of truth for people to recognise what is being lampooned. • We use the example of a small reduction in salt consumption to illustrate the point more fully

12. Small reductions in salt • We already know the direction of change from our prior knowledge. We know the effect of large reductions in salt, and we also realise that the relationship will not change sign when the ‘dose’ is reduced. • We also know that salt is very cheap, and that reductions in it will cause savings of future treatment costs. • End of story. Cost effective. Better than that – cost saving and positive health benefit. • But RCTs have been done on this topic. • The effect size on blood pressure is very small at an individual level • Blood pressure is devilishly hard to measure accurately • Keeping people to a lower-salt diet is well-nigh impossible • So the trials were all underpowered, and showed non-significant results in the ‘right’ direction

13. More about salt • A meta-analysis of the trials (n = 6,489) showed: • Cardiovascular morbidity in people with normal blood pressure (longest follow-up RR 0.71, 95% CI: 0.42 to 1.20, 200 events) or raised blood pressure at baseline (end of trial RR 0.84, 95% CI: 0.57 to 1.23, 93 events) • This prompted a Cochrane press release, 5 July 2011 • Moderate reductions in the amount of salt people eat doesn’t reduce their likelihood of dying or experiencing cardiovascular disease. This is the main conclusion from a systematic review published in the latest edition of the Cochrane Library. • The study has since been withdrawn

14. The currently-used paradigm • We establish an effect by conducting an RCT. The hypothesis-testing approach uses a frequentist paradigm and rules out chance by means of a t-test or similar using p-values. • If significance is achieved, a cost effectiveness analysis is conducted as a second stage. This uses a decision-theoretic paradigm. It establishes cost effectiveness by looking at the size of the estimated mean ICER, and does not consider the ICER variance.

15. Decision theory versus frequentist hypothesis-testing Decision theory • Subjective probability • Prior beliefs • To maximise, variance of effect estimate is ignored • This is as if the decision-maker is risk-neutral • Makes sense if a large number of independent projects are considered. • Used routinely in the business world for maximising profits • Frequentist • Objective probability • No prior beliefs • Does not maximise, and is rather conservative • Decision-maker is risk-averse • Does not consider other projects (maybe shouldn’t if health is concerned?) • Used routinely in medical research for effectiveness.

16. The effect of bias • Decision theory and frequentist hypothesis-testing do not deal well with bias. • The RCT avoids bias internally. A frequentist approach further avoids bias by ignoring prior beliefs. So where bias is important and can be avoided, use a frequentist approach. • The underpowered RCT is very prone to publication bias. RCTs will often be underpowered in public health when the individual effect size is very small. Prior beliefs, however, will often not be subject to much bias, especially if they are firmly held. So use decision theory for effectiveness in PH As in TA

17. Technology appraisal RCT: Estimated distribution of the mean effect Posterior distribution combining prior and RCT Vague prior belief about mean effectiveness • Posterior distribution is influenced almost entirely by the RCT • If the prior distribution is biased, it might detract from accuracy and not enhance it. • So the decision-maker may wish to remain with the red curve (RCT only) as the • most accurate.

18. Public health population intervention Non-vague prior belief about mean effectiveness Posterior distribution combining prior and RCT RCT: Estimated distribution of the mean effect • Posterior distribution is influenced almost entirely by the prior belief curve. • The RCT is mostly a distraction, and it is likely to be biased upwards because of • publication bias • So the decision-maker may wish to ignore the red curve (RCT) because it is not • helpful for the decision-maker. Because we know only the direction of change • with the yellow curve, we must use “what if” analysis to inform cost effectiveness. • To get an effect size (yellow curve doesn’t give it) we use estimated mean RCT effect as an upper bound, and do ‘what-if?’ analysis

19. More on a full decision-theory approach • We have published one paper on this topic (JPH Dec 2013) and have 2 more under submission, and a fourth paper is being written. • NICE processes already reflect this approach, but are not given the gravitas that accords to a fully accepted paradigm like decision theory. (“We must not allow special pleading for Public Health. If the interventions do not satisfy 95% confidence intervals, we cannot accept them.” This comes from the strictest adherents to the frequentist school, but those are the voices that are the most respected.) • In round figures, we believe that the new approach could lead to the same health gains at a cost about £10 billion lower than by the next-best means.

20. Public Goods • The most cost effective interventions in public health are usually some form of public good. • (A “public good” is like a Freeview TV programme – once it is provided to one person, everyone else in the country with a TV set can access it without cost, and cannot be excluded from it. Programmes that are pay-per-view are ‘excludable’, so are not public goods.) • Legislation • Smoking ban (big health gains, low cost of initiating, subsequent cost saving) • Taxation • Almost no work done to implement, improves by reducing drinking/smoking • Regulation • Speed cameras, traffic lights on foodstuffs

21. Public goods (2) • Can’t often do RCTs in public goods, so in the past they have been difficult to model for cost effectiveness. • But recommendations about public goods are telling central government what to do, and central government feels it has primacy when it comes to decision-making. • NICE has to tread carefully in this area and therefore its recommendations must be oblique. • Additionally, health gain is not the only consideration. A recommendation for better health could be to tax beer until a pint in a pub was £20. But this has a political dimension, too, and no politician wishing for re-election could approve such a recommendation. • By making recs that are not the most cost effective, does NICE ‘crowd out’ the most cost effective interventions?

22. Infectious disease modelling • Time horizon • Might be longer than a single lifetime • But models only go for perhaps 20 years • Why? Is it because errors accumulate? • Yet health gains for an individual are usually at the end of life, which might be more than 20 years away. • So the transmission model stops at 20 years, and we then follow through the lives of those in the model till death? Is that the way to do it? • Models are tricky to calibrate • A modelling assignment cannot be done in 3 to 6 months unless modellers have already been working in the area for a couple of years • Much intellectual capital invested. • NICE are required by law to make an executable model available to consultees if requested. This is at odds with the intellectual property rights of the model’s developers. • Maybe there are better ways of ensuring model transparency

23. Infectious disease modelling (2) • Model transparency through calibration • Models are deemed ‘fit’ if they fit the past, and other tests of robustness. • Optimising over time in an infectious disease model would appear to be the finding of an optimal time path out of the infinity of possible paths. • This is an optimal control problem. Has it been modelled as such? Is it optimal to spend a huge amount today in order to keep expenditure low in future, or to spend less now and rather more later? • This is a very tricky problem for governments whose time horizons might be short. NICE might not be the best vehicle for advising government on this.

24. Public health and local government • Public health is now the responsibility of local government. • They have many other objectives, not just health. • In particular, they spend a lot of money on caring for the elderly, which means additional expenditure and incentives for preventing dependency in old age. • The QALY has far less meaning, and probably the only common currency for evaluation studies is money. • They have far tighter budgets than central government, and thus their time horizons are shorter/their discount rates are higher. • The electoral cycle is only 3 years, again raising the discount rate

25. Conclusions • Many rethinks need to be carried out • A change to a decision-theoretic from a frequentist paradigm • Learning more about transmission models and how they can best be used for determining cost effectiveness • A more pragmatic approach to decision-making in line with local government priorities and capabilities • An educative role carried out by modellers and health economists for both local councillors and LG officials • There’s a tension between the immediacy of treating the sick and ensuring in the longer term that the well do not become sick. That is an implied SVJ that will probably always be with us.

26. Infectious disease modelling (3) • In my past life as an experimental economist, I played with the idea of a vaccination programme that allowed two optimal societal vaccination schedules. • At a lower level of vaccination, there was a local optimum that kept a disease under control • At a higher level of vaccination, there was a global optimum (that by definition gave higher benefits to society than the local optimum) which eradicated the disease. • In my replications of the model, the global optimum was not attained, because of free riders.