Kyle J. Myers, Ph.D. Director, Division of Imaging, Diagnostics, and Software Reliability

1. FDA's role in the innovation and evaluation of evolving computer-aided diagnosis (CAD) solutions Kyle J. Myers, Ph.D. Director, Division of Imaging, Diagnostics, and Software Reliability Office of Science and Engineering Laboratories Center for Devices and Radiological Health U.S. Food and Drug Administration

2. FDA Regulates a Spectrum of Health Products: 20-25 cents of every U.S. GDP dollar Animal Drugs, Devices, Pet Foods Foods and Cosmetics Tobacco Products Vaccines Pharmaceuticals Medical Devices Blood Products www.fda.gov

3. CDRH Mission • The mission of the Center for Devices and Radiological Health (CDRH) is to protect and promote the public health.  • We assure that patients and providers have timely and continued access to safe, effective, and high-quality medical devices and safe radiation-emitting products.  • We provide consumers, patients, their caregivers, and providers with understandable and accessible science-based information about the products we oversee.  • We facilitate medical device innovation by advancing regulatory science, providing industry with predictable, consistent, transparent, and efficient regulatory pathways, and assuring consumer confidence in devices marketed in the U.S. www.fda.gov

4. Outline • Medical device premarket submissions • Research activities to support innovation and premarket decision-making for devices • New FDA initiatives www.fda.gov

5. What is a Medical Device? Regulated under authority of Federal Food, Drug, and Cosmetic Act (Section 201h) • “… an instrument, apparatus, implement, machine, … intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment or prevention of disease, in man or other animals, or intended to affect the structure or any function of the body of man or other animals …” As simple as a tongue depressor. As complex as a fully implantable artificial heart or a robotic surgery system www.fda.gov

6. Medical Device Classification • Intended Use / Indications for Use • Risk-Based Paradigm • Medical devices are classified and regulated according to their degree of risk to the public Low Risk High Risk Class I Class II Class III www.fda.gov

7. Premarket Submissions • Premarket notification or 510(k) • Moderate risk device where special controls can mitigate the risks to health • Sponsor must demonstrate that the device is as safe and effective as a legally marketed device (the predicate) • Premarket approval or PMA • Benefits outweigh the risks • De novo • Establishes pathway for innovative low-to-moderate risk devices (no predicate)

8. Special Controls • Generally device-specific • Examples: • Premarket data requirements • Performance standards • Special labeling requirements • Post-market surveillance www.fda.gov

9. Substantial Equivalence • Option 1 • Has the same intended use as the predicate; and • Has the same technological characteristics as the predicate; www.fda.gov

10. Substantial Equivalence • Option 2 • Has the same intended use as the predicate; and • Has different technological characteristics and the information submitted to FDA, including appropriate clinical or scientific data where necessary, demonstrates that the device: • Does not raise different questions of safety and effectiveness than the predicate; and • Appropriate clinical or scientific data demonstrate that the device is at least as safe and effective as the predicate www.fda.gov

11. Summaries of S and E are posted online https://www.accessdata.fda.gov/cdrh_docs/pdf8/K083423.pdf www.fda.gov

12. Summaries of S and E are posted online https://www.accessdata.fda.gov/cdrh_docs/pdf8/K083423.pdf www.fda.gov

13. Intended use Intended use – purpose of the device or its function. The intended use of a device encompasses the indications for use Indications for use – The disease or condition the device will diagnose, treat, prevent, cure or mitigate, including a description of the patient population for which the device is intended

14. General intended uses Intended use of many imaging devices is very general “..intended to produce cross-sectional images of the body by computer reconstruction of x-ray transmission data…” “… general purposed ultrasound imaging and analysis systems providing digital acquisition, processing and display capability and clinical applications including: Abdominal, Obstetrical, …” “…a diagnostic imaging modality that produces cross-sectional transaxial, coronal, sagittal, and oblique images that display anatomic structures of the head or body…These images and/or spectra when interpreted by a trained physician yield information that may assist in diagnosis.” 14

15. Diagnostic radiological devices CDRH clears most diagnostic imaging devices through the 510(k) pathway, including ultrasonic pulsed doppler imaging systems, magnetic resonance diagnostic devices, computed tomography x-ray systems, full-field digital mammography, and picture archiving & communications systems 15

16. 21 CFR 864.5260 Automated cell-locating devices Examples Automated hematology analyzers (differential cell counters) Chromosome analyzers FISH enumeration systems Urine sediment analyzers Image analysis in pathology www.fda.gov

17. Intended Use “This device provides tools for measuring tumor volume.” is not the same as “The device is a Computer-Aided Detection (CAD) system designed to assist radiologists in the detection of solid pulmonary nodules during review of CT scans of the chest.” or “This device may be used to assess/measure changes in tumor volume greater than 25 mm3 from CT images in patients with lung cancer.” 17

18. Statements of this sort: “this device enables estimation of cancer risk”… or “prediction of tumor response to therapy”… or “classification/separation of patient groups”… would need data supporting such statements 18

19. Premarket Approval (PMA) • Class III devices • Demonstrate reasonable assurance of safetyand effectivenessfor its intended use based on sufficient scientific evidence • Very device specific • Standalone submission • No comparison to a predicate • When FDA finds a device safe and effective for its intended use, it is “approved” www.fda.gov

20. Examples of PMA devices https://breast360.org/en/topics/2016/02/17/screening-whole-breast-ultrasound/ Digital Breast Tomosynthesis (DBT) systems for screening High Intensity Focused Ultrasound (HIFU) Mammography Computer-Aided Detection Devices 20 Automated Breast Ultrasound Systems for screening of women with dense breasts

21. Gynecologic Cytology Imaging Systems Cytyc/HologicThinPrep Imaging System Becton Dickinson/TriPathFocalPoint Guided Screening System PapanicolaouStain Detection algorithm, neural network PMAs for pathology CAD devices www.fda.gov

22. De Novo • Novel devices that have not previously been classified are Class III by default (and hence, PMA devices) • De novo is a petition for down classification (Class III to Class II or Class I) • De novo petition must propose controls that would be needed to assure the safety and effectiveness of the device www.fda.gov

23. Specific intended use “intended to measure liver iron concentration to aid in the identification and monitoring of non-transfusion-dependent thalassemia patients receiving therapy with deferasirox” 23

24. De Novo • A granted de novo establishes a new device type, a new device classification, a new regulation, and necessary general (and special) controls • Once the de novo is granted, the device is eligible to serve as a predicate • All the followers are 510(k) devices 24 www.fda.gov

25. Computer-Aided Detection • First devices approved in 1998 • Mammographic CADe • Automated PAP smear reader (up to 25% of slides determined to be normal with no need for further review) • Caries detector • Additional application areas • Chest x-ray • Lung CT • Colon CTC • Quantitative imaging applications, e.g., HeartFlow www.fda.gov

26. General device information • Technological characteristics • Algorithm design and function • Processing steps • Features • Models and classifiers • Training paradigm • Databases • Reference standard • Scoring methodology • Assessment www.fda.gov

27. Technological Characteristics • Needed for • Understanding scientific basis for device safety and performance • Comparing two devices • May reduce performance testing requirements • Understanding scope of a change in a modified CAD algorithm www.fda.gov

28. Algorithm Design and Function • Flowchart describing processing, features, models, classifiers • How are algorithm parameters selected? Better description increases confidence in the device www.fda.gov

29. Databases Normals With disease • Patient data • Demographics • How the imaging data were collected • Collection sites • Number of cases, stratification • Comparison of characteristics of the patient data to the target population • History of accrual and use in both training and testing • Phantom data and simulations can also be scientific evidence www.fda.gov

30. Reference Standard • How is disease presence/absence determined for the cases in the sponsor’s database? • An established clinical determination (e.g., biopsy) • A follow-up examination • Output from another device • An interpretation by reviewing clinicians (truthers) • number and qualifications of truthers • specific criteria used as part of truthing • other clinical information utilized Miller SPIE 2004 v5372 www.fda.gov

31. Reference Standard • How is disease presence/absence determined for the cases in the sponsor’s database? • An established clinical determination (e.g., biopsy) • A follow-up examination • Output from another device • An interpretation by reviewing clinicians (truthers) • number and qualifications of truthers • specific criteria used as part of truthing • other clinical information utilized First CT Lung CAD used panel truth in determination of actionability. Variability in truth was assessed via resampling. Miller SPIE 2004 v5372 www.fda.gov

32. Standalone Assessment • Report the performance of the CAD device by itself, in the absence of any interaction with a clinician • Evaluate performance on different subsets • Acquisition parameters • Lesion properties • Size, severity, etc. • Powering for subsets is not necessary unless specific claims are desired www.fda.gov

33. Scoring Methodology • Correspondence between CAD output and reference standard • Example: • Device TP • CAD mark within a certain distance from the reference standard • Device FP • CAD mark away from the reference standard • Scientific justification www.fda.gov

34. Training to the Test • If you get to interrogate the test set many times, your so-called “test” set becomes part of training • Performance seems to improve for the so-called “test” set as the device adapts • If you collect a fresh dataset from the population, you can find that performance has actually declined Resubstitution bias Number of features Limited-training bias www.fda.gov

35. Test Data Reuse • Permitted with tight controls: • Data randomly selected from a larger database that grows over time • Fixed limit on the number of times a case can be used for testing • Tight control on test data access: • Algorithm development team does not have access to test data • Only summary performance results are reported outside of the assessment team • Data access log is maintained www.fda.gov

36. Clinical Assessment 510(k), PMA, or De Novo • Why needed: • Reader is an integral part of the diagnostic process for CAD devices • Comparison of overall standalone performance may not be adequate when • Substantive technological differences with predicate, resulting in marks with substantially different characteristics • Differences in indications for use Jiang et al., Acad. Radiol. 1999 www.fda.gov

37. Guidance on Clinical Assessment Multi-reader, multi-case (MRMC) study protocol Statistical analysis plan Subset analysis www.fda.gov

38. Adaptive Devices • Device algorithm evolves over time based on the new data collected in the field after the device goes on the market www.fda.gov

39. Current Guiding Principles for Changes in 510(k) Devices • If a manufacturer modifies their device with the intent to significantly improve the safety or effectiveness of the device a new 510(k) is likely required • Software change guidance • Need new 510(k) if • Software change could significantly affect clinical functionality or performance specifications directly associated with intended use http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM514737.pdf www.fda.gov

40. Questions Regarding Adaptive Devices • How can we ensure that the changes do not compromise the safety and effectiveness of the device? • Risk associated with the device • Extent of automated adaptation • Can and should performance testing be conducted after each algorithm modification? Less often? • Test dataset reuse? Device performance over time t0 tn www.fda.gov

41. PreSubmissions (Qsubs) • The FDA/CDRH pre-submission program allows manufacturers to request feedback from the FDA on their proposed regulatory pathway and test protocols • Helps avoid delays in device submission or repeating clinical studies • Encouraged for CAD applications with novel or specific intended uses • Guidance: Request for Feedback on Medical Device Submissions http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM311176.pdf 41 www.fda.gov

42. Example Pre Submission Questions • Does FDA agree that the proposed test plan is sufficient to support the stated indication? • Is FDA aware of standardized test methods to address concern XYZ? • FDA has requested that we do X; would the proposed combination of Y and Z be acceptable to the Agency? www.fda.gov

43. FDA's research in support of innovation of CAD solutions

44. Office of Science and Engineering Labs • Conducts laboratory-based regulatory research to facilitate the development and innovation of safe and effective medical devices • Provides expertise, data, and analyses to support regulatory process • Collaborateswith academia, industry, government, and standards development organizations to develop, translate, and disseminate information regarding regulated products • AAPM, MITA, QIBA/RSNA, IEC, Whole Slide Imaging Working Group, Medical Device Innovation Consortium,… www.fda.gov

45. Imaging Clinical Trials – Statistical Assessment Methodologies • Adaptive study designs • Alternatives to ROC • Precision/Recall • Agreement, C-indices • Utility • Calibration of classification algorithms to a risk probability • Designing, executing, and analyzing multi-reader imaging studies www.fda.gov

46. Calibration of algorithm outputs Uncalibrated Calibrated • Transforming scores on an arbitrary scale to outputs that are calibrated and comparable across systems • A device output of x% means that x of 100 patients actually have the outcome • Three calibration methods • Uncertainty estimation • Simulation studies Device output Device output • Under rationality assumption, calibration is independent of discrimination Chen et al., Stat. Meth. Med. Res., 2016

47. Precision-Recall Curve • Alternative performance evaluation technique • Information retrieval systems • Content-based image retrieval systems • AUCPR: Area under the precision-recall curve • Limited work in investigating variance Giger et al, SPIE Medical Imaging 2002 4684:768-773 www.fda.gov

48. AUCPR • Proposed a new method for estimating AUCPR • Inherits many nice features of semiparametric estimation of the area under the ROC curve • Can be used for system assessment and statistical comparison www.fda.gov Sahiner et al., SPIE Medical Imaging 2016 97870D; 2017-10136-15

49. Alternatives to clinical trials • Physical phantoms • Hybrid data sets • Modeling and simulation www.fda.gov

50. Well-controlled phantom studies to facilitate assessment of impact of image acquisition and analysis procedures on quantitative imaging performance. Public Release of FDA Phantom Data Color phantom for Whole-Slide Imaging CT of phantom with embedded nodules Liver phantom with lesion inserts Coronary artery phantom 50