USE OF EVIDENCE IN DECISION MODELS: An appraisal of health technology assessments in the UK

1. USE OF EVIDENCE IN DECISION MODELS: An appraisal of health technology assessments in the UK Nicola Cooper Centre for Biostatistics & Genetic Epidemiology, Department of Health Sciences, University of Leicester, U.K. http://www.hs.le.ac.uk/personal/njc21/ Acknowledgements to: Doug Coyle, Keith Abrams, Miranda Mugford & Alex Sutton

2. BACKGROUND • Increasingly decision models developed to inform complex clinical/economic decisions (e.g. NICE technology appraisals). Technique commonly used for: • i) Extrapolation of primary data beyond endpoint of a trial, • ii) Indirect comparisons when no ‘head-to-head’ trials • iii) Investigation of how cost-effectiveness of clinical strategies/interventions changes with values of key parameters • iv) Linking intermediate endpoints to ultimate measures of health gain (e.g. QALYs) • v) Incorporation of country specific data relating to disease history and management.

3. USE OF EVIDENCE IN HTA DECISION MODELS (Cooper et al, In press) • OBJECTIVE: • Review sources & quality of evidence used in the development of economic decision models in health technology assessments in the UK • METHODOLOGY: • Review included all economic decision models developed as part of the NHS Research & Development Health Technology Assessment (HTA) Programme between 1997 and 2003 inclusively. • Quality of evidence was assessed using a hierarchy of data sources developed for economic analyses (Coyle & Lee 2002) & good practice guidelines (Philips et al 2004).

4. GOOD PRACTICE CRITERIA FOR DECISION MODELS (Philips et al 2004) • Statement of perspective • Description of strategies/comparators • Diagram of model/disease pathways • Development of model structure and assumptions discussed • Table of model input parameters presented • Source of parameters clearly stated • Model parameters expressed as distributions • Discussion of model assumptions • Sensitivity analysis performed • Key drivers/influential parameters identified • Evaluation of internal consistency undertaken

5. HIERARCHY OF DATA SOURCES • Hierarchy of evidence - a list of potential sources of evidence for each data component of interest: • Main clinical effectiveness • Baseline clinical data • Adverse events and complications • Resource use • Costs • Utilities • Sources ranked on increasing scale from 1 to 6, most appropriate (best quality) assigned a rank of 1

6. HIERARCHY OF DATA SOURCES #Surrogate outcomes = an endpoint measured in lieu of some other so-called true endpoint (including survival at end of clinical trial as predictor of lifetime survival)

8. #One HTA reported both decision & Markov models, one reported both Markov & Individual Patient models, and one model type was unclear. 147 out of 180 (73%) considered Health Economics 48 out of 147 (33%) Developed Decision Models 6 out of 48 (15%) Cost Analyses Models 42 out of 48 (88%) Economic Evaluation Models 26 out of 42 (62%) Decision Trees# 12 out of 42 (29%) Markov Models# 5 out of 42 (12%) Individual Sampling# FLOW DIAGRAM 180 HTA published 1997-2003 22 (out of 42) NICE Appraisals

9. GOOD PRACTICE CRITERIA FOR DECISION MODELS (n=42)

10. RESULTS FROM APPLYING HIERARCHIES OF EVIDENCE (n=42 decision models)

11. Rank 1 Rank 2 Rank 3 Rank 4 Unclear N/A Rank 5 Rank 6 High Medium low

12. CONCLUSIONS • Evidence on main clinical effect mostly: • identified & quality assessed (76%) as part of companion systematic review for HTA • reported in a fairly transparent & reproducible way. • For all other model inputs (i.e. adverse events, baseline clinical data, resource use, and utilities) • search strategies for identifying relevant evidence rarely made explicit • sources of specific evidence not always reported

13. CONCLUSIONS • Concerns about decision models confirmed by this study: • (1) Use of data from diverse sources (e.g. RCTs, observational studies, expert opinion) - may be subject to varying degrees of bias due to confounding variables, patient selection, or methods of analysis • (2) Lack of transparency regarding identification of model input data & key assumptions underlying model structure and evaluation • (3) Bias introduced by the researcher with regards to choice of model structure & selection of parameter values to input into the model.

14. CONCLUSIONS • Hierarchies of evidence for different data components provide useful tool for assessing • i) quality of evidence, • ii) promoting transparency, & • iii) informing weakest aspects of model for future work. • Acknowledged, highly ranked evidence for certain model parameters may not always be available . • Value of evidence input into decision models, regardless of position in hierarchy, depends on its quality & relevance to question of interest. • QUANTITYvs.QUALITY (PRECISIONvs.BIAS)

15. UNANSWERED QUESTIONS • How best to identify the relevant evidence? • How much evidence is sufficient and when would there be benefit from identifying additional/supplementary evidence? • How to appropriately assess, and where possible adjust for, quality of different types of evidence? • - Instruments for assessing quality within study designs but across different study designs non-trivial (Downs & Black 1998) • How to appropriately combine/synthesis evidence from different study types? For example, • meta-analyse all data assuming equal weight, • observational data as prior for RCT data, or • hierarchical synthesis model

16. REFERENCES • Cooper NJ, Coyle D, Abrams KR, Mugford M, Sutton AJ. Use of evidence in decision models: An appraisal of health technology assessments in the UK to date.Journal of Health Services Research and Policy (In press 2005). • Coyle D, Lee KM. Evidence-based economic evaluation: how the use of different data sources can impact results. Donaldson C, Mugford M, Vale L. Evidence-based health economics: From effectiveness to efficiency in systematic review. London: BMJ Publishing Group, 2002: 55-66. • Philips Z, Ginnelly L, Sculpher M et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technology Assessment.2004; 8(36). Copy of slides available at: http://www.hs.le.ac.uk/personal/njc21/