References

1. BRIDGING MULTICRITERIA DECISION ANALYSIS (MCDA) WITH HEALTH TECHNOLOGY ASSESSMENT (HTA) FOR POLICY AND CLINICAL DECISIONMAKING: CASE STUDIES IN CANADA AND SOUTH AFRICA Mireille M Goetghebeur PhD,1,2 Cheri Deal MD PhD,2,3 Jacqui Miot PhD,4 Michele Tony MSc,1* Monika Wagner PhD,1 Hanane Khoury PhD,1 Donna Rindress PhD,1 Tina Papastavros PharmD,5 Paul Oh MD6 1 BioMedCom Consultants inc, Dorval, Quebec, Canada; 2 Centre Hospitalier Universitaire Ste-Justine, Montréal, Québec, Canada; 3University of Montreal, Montréal, Québec, Canada;4 University of Pretoria, Pretoria, South Africa; 5 Ontario Workplace Safety Insurance Board, Toronto, Ontario, Canada;6 Toronto Rehabilitation Institute, Toronto, Ontario, Canada Expert Panel Canada Growth hormone Private Health Plan South Africa Liquid based cytology Public Health Plan Canada Tramadol Private Health Plan South Africa Expert Panel Canada Public Health Plan Canada D1 Disease severity D1 Disease severity D2 Size of population affected by disease D2 Size of population affected by disease C1 Clinical guidelines C1 Clinical guidelines C2 Comparative interventions limitations C2 Comparative interventions limitations I1 Improvement of efficacy/effectiveness I1 Improvement of efficacy/effectiveness I2 Improvement of safety & tolerability I2 Improvement of safety & tolerability I3 Improvement of patient reported outcomes I3 Improvement of patient reported outcomes T1 Public health interest T1 Public health interest T2 Type of medical service T2 Type of medical service E1 Budget impact on health plan E1 Budget impact on health plan • Background • Healthcare decisionmaking is a complex process that requires consideration of a wide range of scientific and contextual criteria and inherently involves value judgments.1 • In assisting healthcare decisionmaking, Health Technology Assessment (HTA) is evolving to address, in addition to clinical and economic evidence, social, organizational, ethical and legal dimensions of health technology.2-6 • Multicriteria decision analysis (MCDA) structures complex decision problems into a comprehensive set of criteria which allows decisionmakers to systematically and explicitly consider multiple dimensions,1,7-10while clarifying their fundamental objectives and values.11 • Bridging HTA with MCDA, the EVIDEM framework includes a comprehensive set of decision criteria with tools and processes: i) to synthesize evidence for each decision criterion (HTA module), and ii) to clarify the perspectives of decisionmakers and allow for systematic and explicit consideration of each decision criterion (MCDA module).12,13 Results • References • Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost EffResourAlloc 2006;4:14. • INAHTA - International Network of Agencies for Health Technology Assessment. HTA Resources. 2010. http://www.inahta.org/HTA/ . (Accessed 8 Mar 2010). • Velasco GM, Gerhardus A, Rottingen JA, Busse R. Developing Health Technology Assessment to address health care system needs. Health Policy 2010;94(3):196-202. • Johri M, Lehoux P. The great escape? Prospects for regulating access to technology through health technology assessment. Int J Technol Assess Health Care 2003;19(1):179-93. • Lehoux P, Williams-Jones B. Mapping the integration of social and ethical issues in health technology assessment. Int J Technol Assess Health Care 2007;23(1):9-16. • Battista RN. Expanding the scientific basis of health technology assessment: a research agenda for the next decade. Int J Technol Assess Health Care 2006;22(3):275-80. • Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ 2006;15(7):689-96. • Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan 2007;22(3):178-85. • Baltussen R, Youngkong S, Paolucci F, Niessen L. Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health Policy 2010; • Nobre FF, Trotta LT, Gomes LF. Multi-criteria decision making--an approach to setting priorities in health care. Stat Med 1999;18(23):3345-54. • Peacock S, Mitton C, Bate A, McCoy B, Donaldson C. Overcoming barriers to priority setting using interdisciplinary methods. Health Policy 2009;92(2-3):124-32. • Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and Value: Impact on DEcisionMaking - the EVIDEM framework and potential applications. BMC Health Serv Res 2008;8(1):270. • Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmaking framework to growth hormone for Turner syndrome patients. Cost EffResourAlloc 2010;8(1):4. E2 Cost-effectiveness of intervention E2 Cost-effectiveness of intervention E3 Impact on other spending E3 Impact on other spending Q2 Completeness and consistency of reporting evidence Q2 Completeness and consistency of reporting evidence Q3 Relevance and validity of evidence Q3 Relevance and validity of evidence • Importance of criteria varied to a great extent among individual decisionmakers (SDs) and across settings. • In all three settings, “Improvement in efficacy/effectiveness” and “Relevance and validity of evidence” were among the criteria considered most important. • “Clinical guidelines” was among the least important criterion in all 3 case-studies. • Information synthesised for each contextual criterion supported decisionmakers in their reflection on ethical and context-specific considerations • The Contextual tool allowed explicit capture of the perspectives of decisionmakers on contextual factors • In all three settings, a majority of decisionmakers felt that most decision criteria of the framework should be considered systematically, independently of the intervention under scrutiny. • For scientific criteria, greatest variations across settings were observed for “Size of the population affected by the disease” and “Clinical guidelines. • For contextual criteria, largest discrepancies across settings were observed for “Population priority and access - Fairness”, “Stakeholder pressures” and “Political/historical context”. Table 2: Qualitative impacts assigned to contextual criteria by decisionmakers Figure 1: Mean relative weights (SD) assigned to scientific criteria by decisionmakers Table 1: Percentage of decisionmakers reporting that criteria should be considered systematically* • Objective • To field-test the EVIDEM framework and explore its utility for both clinical and health policy decisionmaking. • Methods • Decisionmakers field-tested the framework by selecting case-studies relevant to their settings including: • tramadol for chronic non-cancer pain for a drug coverage advisory committee of a public Canadian health plan (Ontario Workplace Safety Insurance Board [WSIB]); • liquid-based cytology for cervical cancer screening for a coverage advisory committee of a private South African health plan (Discovery Health); • growth hormone for Turner syndrome for Canadian pediatric endocrinologists and other healthcare stakeholders (Expert panel).13 • For each case study, evidence was systematically extracted and synthesized for each decision criterion of the framework (15 scientific decision criteria - MCDA matrix- and 6 contextual decision criteria - qualitative tool); the HTA report thus produced was posted on a web information system. • To test the framework during workshops, decisionmakers first assigned weights to each criterion of the MCDA matrix, a step designed to quantitatively capture individual perspectives. Then, using synthesized evidence, they appraised the selected intervention by assigning scores to each scientific criterion and by designating a type of impact on the appraisal (negative, neutral, positive) to each contextual decision criterion • MCDA value estimates were obtained using a linear model combining normalized weights and scores • Feedback on the instruments and process was collected from decisionmakers via survey and discussion. • Mean scores were considered to be a good representation of the perceived performance of the intervention. • In all settings, greatest inter-rater agreement on scores (smaller SDs) was observed for “Improvement of efficacy/effectiveness”, “Impact on other spending” and “Completeness and consistency of reporting evidence”. • Greatest inter-rater variability of scores (greater SDs) was observed for “Clinical guidelines”, “Cost-effectiveness” and “Comparative interventions limitations”. • A majority of decisionmakers reported that the framework improved: • Consideration of all elements of decision (all 3 settings) • Transparency of the decision (public and private health plans studies) • Comprehension of the decision among stakeholders (private health plan and expert panel studies) • Challenges were pointed out: • Understanding the language and meaning of certain decision criteria • Perception that distilling evidence to populate the framework is more difficult than current process • Current lack of reference points to compare/rank MCDA value estimates of interventions • Acknowledgments • We wish to acknowledge the contributions of the members of: the Drug Advisory Committee of the WSIB, the Clinical Policy Unit of Discovery Health, and the Expert panel (Mary Edwards, Turner Syndrome Society of Canada; Jack Holland, MD, McMaster University; Philip Jacobs, PhD, University of Alberta; Sheila Kelton RN, British Columbia Children’s Hospital; Farid Mahmud, MD, University of Toronto; Shayne Taback, MD, University of Manitoba; and Guy Van Vliet, MD, Ste Justine University Hospital Center).The expert panel study was funded by an unrestricted research grant for Pfizer Canada. For the WSIB and Discovery Health studies, internal sources of support were provided by the WSIB, Discovery Health and BioMedCom. • *Michèle Tony received a MSc grant from BioMedCom for the WSIB Study. Figure 2: Mean scores (SD) assigned to scientific criteria by decisionmakers • Conclusions • There is growing interest in MCDA to promote more explicit and systematic approaches in clinical and policy decisionmaking • The proposed approach allows to capture variations in perspectives across individuals and settings, highlighted in this study by striking differences on the importance of criteria between a panel of experts with a clinical focus and standing advisory committees of health plans • By structuring HTA on decision criteria rather than on data produced, MCDA based approaches represent a paradigm shift for appraisal of healthcare interventions • Further research and testing of MCDA based frameworks such as EVIDEM is essential to collect feedback from users and tackle challenges in implementing these innovative approaches. • Linear combination of normalized weights and scores resulted in committee specific MCDA value estimates of 0.43 (0.36 min – 0.50 max) for liquid based cytology, 0.41 (0.26 min – 0.54 max) for growth hormone and 0.44 (0.36 min – 0.61 max) for tramadol, on a scale of 0 (no value) to 1 (maximum value). *Data obtained from survey questionnaire administered during workshops