Michael E. Matheny, MD, MS NLM Biomedical Informatics Fellow

1. Methodologies and Automated Applications for Post-Marketing Outcomes Surveillance of Medical Devices and Medications Michael E. Matheny, MD, MS NLM Biomedical Informatics Fellow Decision Systems Group, Department of Radiology Brigham & Women’s Hospital, Boston, MA

2. Outline • Post-Marketing Surveillance Background • Statistical Methodology Development • Computer Application Development • Clinical Examples • Future Directions

3. BackgroundSurveillance Rationale • Phase 3 Trials insufficient to ensure adequate safety of medications and devices • Low frequency events are not detected • Protected populations (pregnant women, children) and more ill populations not represented • Complications delayed by a number of years cannot be detected

4. BackgroundFDA Medical Devices • 1,700 types of devices • 500,000 device models • 23,000 manufacturers

5. BackgroundFDA New Drug Applications (NDA)

6. BackgroundCurrent Post-Marketing Surveillance • Combination of mandatory and voluntary adverse event reporting • Mandatory reporting by manufacturers and health facilities • Voluntary MedWatch / MAUDE reports by providers and patients

7. FDA Warning FDA Warning Cancelled BackgroundMAUDE Cypher Reporting Rate 2003 2004

8. BackgroundCurrent Post-Marketing Surveillance • ‘Phase 4’ Trials • Poor Compliance • As of March 2006 report, 797 of 1231 (65%) agreed-upon trials had yet to be started • Barriers • Lack of manufacturer incentives • Expensive • Drug already on the market • Lack of regulatory enforcement

9. BackgroundProduct Recalls • Boston Scientific cardiac stent (1998) • Balloon rupture at low pressures • Guidant cardio-defibrillator (2005) • Malfunction due to electrical short • Vioxx (2004) • cardiovascular complications • Tequin (2006) • Hypoglycemia and hyperglycemia

10. BackgroundFDA Response • Increasing demands for Phase 4 trials • Legislation to increase quality of adverse event reporting • Emphasizing trial registries (clinicaltrials.gov) as way to prevent omission of results • Commissioned IOM report “The Future of Drug Safety”

11. BackgroundAdverse Event Data Continuum

12. Statistical MethodsMedical Outcomes Monitoring • Using registry data that tracks all patients allows different types of analysis than used in the FDA’s adverse event reporting systems • No generally accepted methods for monitoring registry data for adverse events • Lack of sufficient discrete electronic data sources to construct registries • Some outcomes are challenging or expensive to track for an entire population

13. Objective • Develop methodologies and implement an automated computer monitoring system to perform outcomes surveillance of registry data for new medical devices and medications

14. Statistical MethodsStatistical Process Control

15. Statistical MethodsBayesian Updating Statistics

16. Statistical MethodsEstablishing Baseline Data • Primary Data Sources • Phase 3 trial data • Post-Marketing data from a closely related medication/device • Alternative Data Sources

17. Statistical MethodsEstablishing Alerting Thresholds • Use number of events and sample size to calculate proportion with confidence intervals • Typically, medical domains use 95% CI or 1.96 sigma from the point estimate

18. Statistical MethodsEstablishing Alerting Thresholds SPC BUS

19. Statistical MethodsEstablishing Alerting Thresholds

20. Statistical MethodsEstablishing Alerting Thresholds • Wilson’s method of comparison between two proportions

21. Statistical MethodsRisk Stratification • Allows creating subgroups for separate analyses • Single variable • Logistic regression model with scoring thresholds

22. Application DevelopmentDELTA • Data Extraction and Longitudinal Time Analysis (DELTA) • Design Goals • Generic data import format • Allow both prospective and retrospective analyses • Modular framework to allow sequential addition of statistical methodologies • Multiple alerting methods • Any number of concurrent ongoing analyses

23. SPC BUS VPN Intranet Application DevelopmentDELTA Source Database DELTA Database Clinical Data Entry Data Dictionary Web Server Statistical Modules Source IT Manager DELTA Users

24. Application Example DataCypher Drug-Eluting Stent (DES) • Setting: • Brigham & Women’s Hospital (07/2003 – 12/2004) • Population: • All patients receiving a drug-eluting stent (2270) • Outcome: • Post-procedural in-hospital mortality (27) • Baseline: • University of Michigan Data (1997-1999)

25. ApplicationAnalysis Configuration

26. ApplicationAnalysis Configuration

27. ApplicationAnalysis Configuration

28. ApplicationAnalysis Configuration

29. ApplicationAnalysis Configuration

30. ApplicationResults Display

31. ApplicationResults Display

32. ApplicationResults (Moderate Risk Stratum)

33. ApplicationResults (High Risk Stratum)

34. Risk StratificationPotential Solution • Incorporate individual risk prediction models in order to adjust for case mix and illness severity

35. Possible Risk Prediction Methods • Linear / Logistic Regression • Artificial Neural Networks • Bayesian Networks • Support Vector Machines

36. LR External Validation Models

37. LR External Validation • Setting: • Brigham & Women’s Hospital (01/2002 – 09/2004) • Population: • All patients undergoing percutaneous coronary intervention (5216) • Outcome: • Post-procedural in-hospital mortality (71)

38. LR External Validation Results

39. LR External Validation Conclusions • Excellent discrimination across all models • Calibration (Hosmer-Lemeshow) poor for all models but recent local one • Addressed categorical risk stratification by keeping all records in one stratum • Calibration problems over time limit application, and require exploration of recalibration methods

40. OPUS (TIMI-16) • Setting: • 888 Hospitals in 27 Countries • Intervention: • Oral IIb-IIIa Inhibitor vs Placebo • Population: • Intervention Arm [Both arms identical at 30 days] (6867) • Outcome: • 30 day mortality • Trial stopped early due to elevation in intervention arm • Baseline: • Control Arm (3421)

41. OPUS (TIMI-16)30 Day Mortality

42. OPUS (TIMI-16) Alert Summary

43. CLARITY (TIMI-28) • Setting: • 313 Hospitals in 23 Countries • Intervention: • Oral Anti-Platelet Agent vs Placebo • Population • Intervention Arm (1751) • Outcome: • Major Bleeding • DSMB concerned, but trial did not stop early • Baseline • Control Arm (1739)

44. CLARITY (TIMI-28)Major Bleeding

45. CLARITY (TIMI-28)Major Bleeding

46. CLARITY (TIMI-28)Alert Summary

47. OPUS /CLARITYConclusions • SPC performed well in the positive study, but did have some false positive alerts in the negative study • LR stratified SPC failed to alert early in the positive study, but performed well in the negative study • BUS was more specific than SPC in both studies

48. Sensitivity Analysis • Setting: • Brigham & Women’s Hospital (01/2002 – 12/2004) • Population: • All patients undergoing percutaneous coronary intervention (6175) • Outcome: • Post-procedural major adverse cardiac events (403) • Death • Post-Procedural Myocardial Infarction • Repeat Vascularization • Baseline: • Arbitrarily set event rates and sample sizes

49. Sensitivity AnalysisResults

50. Clinical Alert • Setting: • Brigham & Women’s Hospital (01/2002 – 12/2004) • Population: • All patients receiving a vascular closure device after percutaneous coronary intervention (3947) • Outcome: • Retroperitoneal Hemorrhage (25) • Baseline: • Stanford University Data (2000 – 2004)