Nathalie Dourdin PhD, Monika Wagner PhD, Peter Melnyk PhD and Donna Rindress PhD

1. Can decisionmakers rely on health economic evaluations? A review of studies assessing the quality of published health economic evaluations Nathalie Dourdin PhD, Monika Wagner PhD, Peter Melnyk PhD and Donna Rindress PhD BioMedCom Consultants inc., Montréal, QC, Canada BioMedCom, Pure and Applied Intelligence BioMedCom, L’intelligence en action • Issues pertaining to the quality of published HE evaluations were identified by determining how many QA studies reported a given issue as a problem in 20% or more of the HE evaluations they reviewed. • When a study mentioned a quality criterion as an issue, the study was included in the percentage calculation regardless of the reason for mention (e.g. not clear, not stated, not justified, etc.,). • The proportions of QA studies that identified a specific quality issue were then calculated. • Quality issues were grouped in clusters as follows: • Outcome measures, valuation of health outcomes • Types of costs and resources included and year of costing • Methodologies for modeling, validation of model, assumptions • Sensitivity analyses and CEAC (cost effectiveness acceptability curve) • Incremental analyses, calculation errors, wrongful use of efficacy/effectiveness • Potential biases and limitations • Sources of funding and disclosure of potential conflicts of interest • BACKGROUND & OBJECTIVE • Health economic (HE) evaluations are intended to provide decisionmakers with information on the comparative efficiency of medical technologies. • The quality of an HE study is critical to the useful application of its findings. • There are a number of different guidelines and checklists available in the public domain on the conduct and reporting of HE and outcomes research. • Adoption of guidelines should help ensure a high degree of rigour and standardization among HE evaluations. • There are currently 74 guidelines for the conduct and reporting of HE evaluations (published and public domain). • The objective of this work is to review studies assessing the quality of published HE evaluations (QA studies) and to investigate whether quality has improved over time with the use of guidelines. • The most commonly reported quality criteria with issues cited by QA studies were: • Perspective • Sensitivity analyses/uncertainty • Discounting • Types of costs included • Source of funding/conflict of interest • Among 68 studies included, only 15 reported on the change in quality of HE evaluations following guideline publication. • 11found evidence that the quality of studies is improving • 4found no improvement over time and sub-optimal adherence to guidelines • CONCLUSION • Despite a wealth of guidelines, evidence for their impact on the quality of published health economic studies is mixed. • Some reviews of health economic evaluations found evidence that the quality of studies is improving. • Other reviews found no or only modest improvement over time and sub-optimal adherence to guidelines. • Outcome measures for quality of HE studies varied significantly across all these studies and were usually qualitative in nature. • FUTURE DIRECTIONS • We need standardized guidelines to promote more consistent level of quality of HE publications. • This is not sufficient, use and reinforcement by journals publishing HE evaluations (for both reviewers and authors) is needed. • We also need more standardized and quantitative measures of quality. • METHODS • Literature searches • Full-text published HE guidelines and studies assessing the quality of HE evaluations were collected from available public domain sources, including PubMed. • The PubMed database was searched from inception using the search terms “health economic guidelines, quality assessment, cost-benefit analysis/standards, cost-effectiveness analysis/standards” • Limits: Meta-Analysis, Practice Guideline, Review, Consensus Development Conference, NIH, Guideline • English, French, German studies selected • Abstracts from recent conferences of ISPOR, SMDM, HTAi, ASHE and iHEA were searched for any relevant abstracts and presentations • Inclusion criteria: reviews of economic evaluations • Data extraction • Information from retrieved QA studies was systematically extracted using a data extraction template • Data extracted included HE study inclusion/exclusion criteria, type of quality assessment tool(s) used, as well as quality assessment results. REFERENCES 1. Campbell H, Briggs A, Buxton M, Kim L, Thompson S. J Health Serv Res Policy. 2007;12(1):11-7. 2. Campbell JD, Spackman DE, Sullivan SD. Allergy. 2008;63(12):1581-92. 3. Colmenero F, Sullivan SD, Palmer JA, Brauer CA, Bungay K, Watkins J, et al. Am J Manag Care. 2007;13(7):401-7. 4. Greenberg D, Earle C, Fang CH, Eldar-Lissai A, Neumann PJ. J Natl Cancer Inst. 2010;102(2):82-8.5. Jefferson T, Demicheli V, Vale L. JAMA. 2002;287(21): 2809-12. 6. Kaplan RM, Groessl EJ. J Consult Clin Psychol. 2002;70(3):482-93. 7. Manuel MR, Chen LM, Caughey AB, Subak LL. GynecolOncol. 2004;93(1):1-8. 8. Neumann PJ, Greenberg D, Olchanski NV, Stone PW, Rosen AB. Value Health. 2005;8(1):3-9. 9. Neumann PJ, Fang CH, Cohen JT. Pharmacoeconomics. 2009;27(10):861-72. 10. Otero HJ, Rybicki FJ, Greenberg D, Neumann PJ. Radiology. 2008;249(3):917-25. 11. Phillips KA, Chen JL. Am J Prev Med. 2002;22(2):98-105. 12. Schwappach DL, Boluarte TA, Suhrcke M. Cost Eff ResourAlloc. 2007;5:5. 13. Schwappach DL, Boluarte TA. BMC Health Serv Res. 2007;7:7. 14. Spiegel BM, Targownik LE, Kanwal F, Derosa V, Dulai GS, Gralnek IM, et al. Gastroenterology. 2004;127(2): 403-11. 15. Stone PW, Schackman BR, Neukermans CP, Olchanski N, Greenberg D, Rosen AB, et al. Lancet Infect Dis. 2005;5(6):383-91. • RESULTS • 68 QA studies met the inclusion criteria; 50 published after 2000 • Studies used various tools for assessing quality as well as different ways of reporting the quality assessment • Given this heterogeneous approach to QA, some issues may not have been identified in specific assessments • Of the studies published after 2000, 39 different approaches were used to measure quality