Disclaimer

1. Using Disproportionality Analysis to Systematically and Simultaneously Detect Safety Signals in AERSJonathan G. Levine, PhDFDA/CDEROffice Of Pharmacoepidemiology & Statistical Science

2. Disclaimer • The views expressed in this presentation are mine, and do not necessarily represent the views of FDA.

3. Overview • Goals of adverse event analysis. • What is AERS? • Limitations of AERS • AERS versus Clinical Trials. • What can we learn from AERS? • Disproportionality Analysis. • Interpreting Disproportionality Scores

4. Goals of Adverse Event Analysis • Adverse event analysis seeks to answer these three questions: • Which drugs or combinations of drugs cause which adverse events? • Which patients are most likely to experience the adverse event? • Is an alternate therapy more or less likely to cause the event? • AERS data cannot by itself answer these questions • AERS data can help to answer these questions

5. AERS Database • Computerized adverse case reporting system • Voluntary reporting by health care workers and the general public. • Mandatory reporting by manufacturers for serious, unexpected events • Adverse event reports • Coded according to the standardized terminology of the Medical Dictionary for Regulatory Activities(MedDRA) • Over 2.5 million voluntary reports from 1968 to the present. • Small number of data elements (drugs, events, age, sex, etc.) • Lots of missing data • Safety issues leading to drug withdrawal from the market have been discovered using the AERS database

6. Limitations of AERS • No denominator • Population rates cannot be estimated. • Differential under-reporting of events can bias results. • Consistent under-reporting for a drug or event does not bias results • Under-reporting that is a function of drug and event does bias results • Drug name errors • Data entry is not done following strict drug naming standards. • Missing data • MedDRA is often too granular • Numerous terms for seizures, strokes, etc. • MedDRA codes are often difficult to aggregate • Higher level groupings may contain an event and its opposite, e.g. hypertension and hypotension. • MedDRA is not comprehensive • A very unusual adverse event may not have a MedDRA term. • Duplicate reports • Multiple manufacturers may submit the same report.

7. AERS versus Clinical Trials • “Representative” population? • Clinical trials: non-random samples, certain types of patients are intentionally excluded. • AERS: non-random sample, no one intentionally excluded. • Generalizability • Clinical trials: results may not apply to other patient populations. • AERS: results apply to the entire patient population (in theory); reporting may be driven by events incompletely addressed in the labeling. • Number of patients studied • Clinical trials: small (hundreds, maybe thousands of patients). • AERS: everyone (in theory…but under-reporting biases and labeling of an event may in practice reduce the number of patients effectively studied).

8. AERS versus Clinical Trials • Ascertainment? • Clinical trials: good ascertainment for serious events with fairly rapid onset and more subtle events that are the focus of monitoring • AERS: under-reporting • Causality • Clinical trials: Randomization makes causality assessment straightforward. • AERS: causality assessment difficult without using external information. ( e.g. good patient descriptions, linked to an objective marker such as digoxin levels, potassium levels, and a correlated EKG and positive dechallenges and rechallenges) • Complexity of research question • Clinical trials: limited by cost and ethics. • AERS: limited by actual drug use and reporting.

9. What can a statistician learn from AERS? • Has adverse event Y been reported in patients taking drug X? • Are reports in AERS containing drug X more likely to also contain adverse event Y? Question: How can we analyze AERS data in order to answer these questions?

10. Disproportionality Analysis Using DuMouchel’s MGPS Method • Calculate observed and expected number of reports for a particular drug-event combination. Observed rate = Number of reports for event X with drug Y Number of reports for drug Y Expected rate = Number of reports for event X in AERS Number of reports in AERS Reporting Ratio (RR)= Observed rate Expected rate • “Shrink” the RR towards 1. The shrunk RR is referred to as the EBGM score. • The amount of shrinkage is a function of the amount of information in AERS about the drug-event combination.

11. Disproportionality Analysis • MGPS done on a “cleaned” version of AERS • Duplicate removal, drug name standardization • All scores are calculated simultaneously. • Expected rates are usually calculated stratifying by age, sex, and report year. • The strata-level expected rates are combined using strata weights proportional to strata size. • A common measure of the importance of a drug-event combination is the lower limit of the 90% credible region (“confidence interval”) for the EBGM, referred to as EB05.

12. What Does an EB05>1 Mean? • An EB05>1 suggests that in AERS there is an association between the drug and the event. • An EB05>1 does not prove that a drug causes an event. • What is a large EB05 value? • A signal having an EB05 > 2 indicates that the D-E is at least twice the expected • In many cases, an EB05 >1, or EB05 >1.5 may be a more useful definition.

13. When Are Findings from Disproportionality Analysis Most Compelling? • Very low background rate in general population, e.g. “growing feathers” • Very low background rate in similar patients • Objective outcome • Any finding must be viewed in context.

14. A. B. Hill not R. A. Fisher(http://www.edwardtufte.com/tufte/hill) • Hill asked: “What aspects of that association should we especially consider before deciding that the most likely interpretation of it is causation?” • Strength. • Consistency • Specificity • Temporality • Biological gradient • Plausibility • Coherence • Experiment • Analogy • Determination of causality requires more than statistics.

15. Does Disproportionality Analysis Work? • MGPS has identified emerging adverse events. • EB05s > 1 are usually seen for known adverse events. • Implausible signals are rarely seen, (but sometimes the event causes the drug; need medical input to determine direction of causality). • Innocent bystanders problem and signal leakage in cases of polypharmacy with independence model are still a problem. As a means to address these issues, Bayesian logistic regression methods are being studied.

16. Conclusions • AERS provides useful information about adverse events. • Clinical trials cannot replace the information provided by AERS. • Biases exist in AERS, and the exact nature of the biases is impossible to determine. • Disproportionality analysis can provide an understanding of the associations between drug-event pairs in AERS. • Disproportionality analysis of AERS cannot by itself determine if there is a causal link between a drug-event pair.

17. Selected References • Almenoff, J. S., W. DuMouchel, A. Kindman, X. Yang and D. M. Fram (2003). "Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting." Pharmacoepidemiology and Drug Safety 12(6): 517-521. • DuMouchel, W., Bayesian data mining in large frequency tables, with an application to the FDA Spontaneous Reporting System. The American Statistician, 1999. 53(3):177-190. • DuMouchel, W. and D. Pregibon, Empirical Bayes screening for multi-item associations. In 7th ACM SigKDD Intl Conference on Knowledge Discovery and Data Mining. 2001. San Francisco: ACM Press. • Fram DM, Almenoff JS, DuMouchel W (2003) Empirical Bayesian Data Mining for Discovering Patterns in Post-Marketing Drug Safety Data Proc. ACMSIGKDD 2003 Intl. Conf. on Knowledge Discovery from Data. • Niu, M.T., D.E. Erwin, and M.M. Braun, Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine, 2001. 19: 4627-37. • O'Neill, R.T. and A. Szarfman, Discussion: Bayesian data mining in large frequency tables, with an application to the FDA Spontaneous Reporting System by William DuMouchel. The American Statistician, 1999. 53(3):190-6. • O'Neill, R.T. and A. Szarfman, Some FDA perspectives on data mining for pediatric safety assessment. Curr Ther Res Clin Exp, 2001. 62:650-663. • Szarfman, A., S.G. Machado, and R.T. O’Neill. Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDA’s Spontaneous Reports Database. Drug Safety, 2002. 25(6): p. 381-392. • van Puijenbroek EP, Diemont WL, van Grootheest K Application of Quantitative Signal Detection in the Dutch Spontaneous Reporting System for Adverse Drug Reactions Drug Safety 2003; 26 (5): 293-301