Su Golder, MRC Fellow in Health Services Research spg3@york.ac.uk

1. Can unpublished data be of value in systematic reviews of adverse effects? Su Golder, MRC Fellow in Health Services Research spg3@york.ac.uk

2. This research is being undertaken as part of an MRC fellowship • The views expressed in this presentation are those of the author and not necessarily those of the MRC • CRD is part of the National Institute for Health Research (NIHR) and is a department of the University of York

3. Introduction • Why adverse effects are important? • Why include adverse effects in systematic reviews? • Does unpublished adverse effects data differ from published adverse effects data? • Should we include unpublished data in systematic reviews of adverse effects?

4. Why adverse effects are important • Definition • ‘A harmful or undesirable outcome that occurs during or after the use of a drug or intervention for which there is at least reasonable possibility of a causal relation’ Chou et al 2010 • Why adverse effects are important • Can be serious – hospitalisation, long-term disability, death (USA: 4th to 6th leading cause of death) • Quality of life, Compliance • Cost (estimates of cost to NHS of £2 billion per year)

5. Why include harmful effects in systematic reviews • Considering only benefits leads to bias • Need to assess benefit/ harm balance • Detailed evaluation of safety needed when: • Narrow margin between benefit and harm (aspirin/ CVD) • A number of equally effective treatments with different safety profiles • When adverse effects cause withdrawal from treatment

6. Methodological Overview of Studies on Adverse Effects • Comparisons of • Industry funded data versus non-industry funded data • Different information sources (such as MEDLINE versus EMBASE) • Different search strategies (such as adverse effect related subheadings, textwords and indexing terms) • Author affiliation (such as clinician or researcher) • Journal impact factor (high versus low) • Country and date of study • Different study designs (such as RCTs versus observational studies) • Unpublished versus published data

7. Introduction: published and unpublished data • If unpublished data differs systematically from published data, systematic review limited to published data may be biased • Significant positive outcomes more likely to be published than nonsignificant outcomes. Hence, systematic reviews based on published literature may overestimate efficacy of an intervention • Impact of including only published adverse effects not been clarified

8. Methods • Search strategy: 10 databases, bibliographies, handsearching key journals, conference proceedings, websources, contacted experts, citation searching • Inclusion criteria: compared adverse effects of healthcare intervention according to publication status • Analysis: • Descriptive comparison for rates and number of cases • Ratio of odds ratio where odds ratio or relative risk presented “pooled risk ratio for the adverse outcome from unpublished data,’’ divided by ‘‘pooled risk ratio for the adverse outcome from published studies.’’

9. Results

10. Availability of unpublished data • Case Reports • Two out of three studies found more unpublished than published case reports of adverse effects (Bennett et al 2005, Tramer et al 1997, Cosmi et al 2000) • Trials • A higher percentage of unpublished trials report adverse effects than published trials (Hemminki 1980)

11. Differences in published versus unpublished data • Case reports • One study of the rank order of adverse effects from published and unpublished case reports found the most frequent adverse effects in published cases to be respiratory and nervous system adverse reactions, and in unpublished cases to be thyroid and skin disorders (Loke et al 2004) • Trials • Five studies reported the risk ratio (RR) from published and unpublished trials (Hemminki 2000, MacLean et al 2003, Ross et al 1997, Wallace et al 2006, Whittington et al 2004)

13. Differences in published versus unpublished data • Case reports • Unpublished case reports will generate a different picture of the relative frequencies of specific adverse effects • Trials • No clear evidence that data on adverse effects from published and unpublished trials differed • Inclusion of unpublished data could provide more precise estimates of adverse effects

14. Future • More Reviews • Increasing number of reviews including adverse effects either as secondary outcome (in addition to effectiveness) or as primary outcome • Better Reporting • CONSORT • Help and support • Cochrane Adverse Effects Methods Group http://aemg.cochrane.org/ • Discussion List http://lists.cochrane.org/mailman/listinfo/aemg

15. Guidance • CRD’s Guidance • Systematic Reviews: CRD’s guidance for undertaking reviews in health care. 2009 http://www.york.ac.uk/inst/crd/index_guidance.htm • Cochrane Handbook • Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]. The Cochrane Collaboration, 2009. Available from www.cochrane-handbook.org

16. Other Guidance • Loke YK, Price D, Herxheimer A. Systematic reviews of adverse effects: framework for a structured approach. BMC Med Res Methodol 2007;7:32 • Chou R, Aronson N, Atkins D, Ismaila AS, Santaguida P, Smith DH, Whitlock E, Wilt TJ, Moher D. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program J Clin Epidemiol 2010 May;63(5):502-12

17. Included Papers • Bennett CL, Nebeker JR, Lyons EA, Samore MH, Feldman MD, McKoy JM, et al. The Research on Adverse Drug Events and Reports (RADAR) project. JAMA. 2005:293:2131-40. • Cosmi B, Castelvetri C, Milandri M, Rubboli A, Conforti A. The evaluation of rare adverse drug events in Cochrane reviews: the incidence of thrombotic thrombocytopenic purpura after ticlopidine plus aspirin for coronary stenting. 8th Annual Cochrane Colloquium Abstracts, October 2000 Cape Town, Africa. • Hemminki E, McPherson K. Value of drug-licensing documents in studying the effect of postmenopausal hormone therapy on cardiovascular disease. Lancet 2000:355:566-9. • Loke YK, Derry S, Aronson JK. A comparison of three different sources of data in assessing the frequencies of adverse reactions to amiodarone. Br J Clin Pharm 2004:57:616-21. • MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG, Southern CE-BPC. How useful are unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epidemiol 2003:56:44-51. • Ross SD, Kupelnick B, Kumashiro M, Arellano FM, Mohanty N, Allen IE. Risk of serious adverse events in hypertensive patients receiving isradipine: a meta-analysis. J Hum Hypertens 1997:11:743-51. • Tramer MR, Moore RA, McQuay HJ. Propofol and bradycardia: causation, frequency and severity. Br J Anaesth 1997:78:642-51. • Wallace AE, Neily J, Weeks WB, Friedman MJ. A cumulative meta-analysis of selective serotonin reuptake inhibitors in pediatric depression: did unpublished studies influence the efficacy/safety debate? J Child Adolesc Psychopharmacol 2006:16:37-58. • Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 2004:363:1341.

18. Publications • Golder S, Loke YK, McIntosh HM. Room for improvement? A survey of the methods used in systematic reviews of adverse effects. BMC Medical Research Methodology 2006;27;6:3. • Golder S, Loke YK. Is there evidence for biased reporting of published adverse effects data in pharmaceutical industry-funded studies? British Journal of Clinical Pharmacology 2008:66(6)767-773. • Golder S, Loke YK. Search strategies to identify information on adverse effects: a systematic review. Journal of the Medical Library Association 2009:97(2):84–92. • Golder S, Loke YK, Bland M. Unpublished data can be of value in systematic reviews of adverse effects: methodological overview. Journal of Clinical Epidemiology 2010;63(10):1071-1081. • Golder S, Loke YK. Sources of information on adverse effects: a systematic review. Health Information and Libraries Journal 2010;27(3):176-90. • Loke YK, Golder SP, Vandenbrouche JP. Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources. Therapeutic Advances in Drug Safety. 2011 [in press]

19. Industry Data • Raw data from industry • No evidence that the numbers of adverse effects from industry and non-industry funded studies differed. • Interpretation and conclusion of industry studies • Bias may be introduced in the interpretation and conclusions of industry funded studies

20. Sources • MEDLINE • MEDLINE retrieved the highest number of relevant references in only four out of thirteen cases: • One compared MEDLINE with IPA • One compared MEDLINE with reference books • Two searched for non-drug interventions • EMBASE vs MEDLINE • In eight out of ten cases searching EMBASE retrieved more relevant references than MEDLINE.

21. Sources • Derwent Drug File vs EMBASE vs MEDLINE • In four out of five cases searching Derwent Drug File retrieved more relevant references than EMBASE or MEDLINE • Industry Submissions • In two out of four cases data retrieved from industry submissions retrieved the highest number of relevant records and in each case many records were unique • International Pharmaceutical Abstracts (IPA) • Searching IPA retrieved either the lowest or joint lowest number of relevant records in five out of seven cases

22. Search strategies • Evaluated generic adverse effects search strategies • Badgett et al 1999 • (ae or co or po or de).fs or CASE REPORT/ and HUMAN/ • Golder et al 2006 • (ae or co or de).fs or (safe or safety or side effect* or undesirable effect* or treatment emergent or tolerability or toxicity or adrs or (adverse adj2 (effect or effects or reaction or reactions or event or events or outcome or outcomes)).ti,ab

23. Search strategies • Evaluated searches for named adverse effects • Wieland et al 2005 • “human” [MESH] AND journal article [pt] AND breast neoplasms [majr:noexp] AND (risk [mh:noexp] OR risk factors [mh:noexp] OR follow-up studies [mh:noexp] OR odds ratio [mh:noexp])

24. Search strategies • Be careful of large numbers of results • High sensitivity can be achieved but with low precision • Search strategies which have been evaluated, achieved sensitivity 97% to 100%, with precision 0.9% to 2.8% (NNR: 111 to 36) • Don’t rely on electronic searches alone • Even using generic adverse effects free text and indexing terms and synonyms for named adverse effect likely to miss relevant articles

25. Other factors • Author affiliation • Surgeons and those working in screening reported less complications • Impact factor • High impact factor journals reported higher rates of adverse effects • Year of publication • No significant difference • Country Setting • No significant difference

26. Study designs • RCTs less heterogeneous but smaller number of participants • Most confidence intervals of the incidence or risk ratio of adverse effects overlap between study designs • Most study designs agree, in terms of finding an increase, decrease or no difference, in adverse effects • No consistent pattern of reporting between study designs • Observational studies may not pose a major threat of bias in the reporting of adverse effects data but caution is still needed