Accepting CAD for Clinical Practice

1. Accepting CAD for Clinical Practice Maryellen L. Giger, Ph.D., FAAPM Professor & Vice-Chair for Basic Science Research Department of Radiology & Committee on Medical Physics Carl J. Vyborny Translational Laboratory for Breast Imaging Research University of Chicago m-giger@uchicago.edu

2. I am representing myself. • My research is supported with grants from NIH, the U.S. Army DOD, the DOE, and the University of Chicago. • Conflict of Interest: I am a stockholder in R2/Hologic, and I receiving royalties from Hologic, GE Medical Systems, MEDIAN Technologies, Riverain Medical, Mitsubishi, and Toshiba. • Conflict of Interest: I am the current President of the American Association of Physicists in Medicine, however, I am NOT representing the AAPM here today.

3. Concern • As a CAD researcher, I am concerned about the timeliness and consistency of the translation of CAD developments to clinical use.

4. Future of Computerized Image Analysis in Medical Decision Making • Computers are increasingly being incorporated into our lives • It is inevitable that they will be incorporated into medical imaging to aid in patient management • What is important: • How to further the progress of CAD research • How to evaluate new CAD devices • How to expedite the process so that they are incorporated into clinical practice in a timely manner • Realization by users that radiologist training with the computer output is crucial when introducing CAD into the clinical arena

5. How to evaluate a new CADe system??? • Least burdensome approach to demonstrate substantial equivalence • Standardization of testing including scoring method and ground truth • Maintenance of the integrity of “test set”, and avoid reuse of data for testing • Case mix: allow for enrichment of a test set with different lesion types, cancer prevalence, etc. to match the clinical context • Reproducibility • Reader mix: Realization that in ultimate clinical practice, the user will change with different CADe systems - so potentially could take user out of the evaluation??

6. Suggest a Two-Stage Method of Evaluation • BACKGROUND: “Independent Double Reading” • Double reading improves detection sensitivity with some increase in recall rate • Occurs if intended goal of double reading is to improve sensitivity • Double reading is accepted (although no FDA requirement) • The increase in sensitivity and recall rate are similar for double read and single read with CAD

7. Suggest a Two-Stage Method of Evaluation • STAGE ONE: • Determine a performance level standard for CAD based on published data and a cooperative study • With academia, government agencies, and commercial CAD manufacturers, demonstrate equivalency between “single read with CAD” and “double read”, [or demonstrate improvement with CADe as compared to no CADe]. • Using radiologists trained in CAD usage and CADe system(s) • Perform the cooperative study as an “ACRIN-like” study • Output from this ONE LARGE READER STUDY will yield the needed “minimum bar” level of a computer alone performance in terms of detection sensitivity and false-positives marks per image.

8. Suggest a Two-Stage Method of EvaluationAfter we know that CADe is useful via the “STAGE ONE” Study • STAGE TWO (a): • For evaluation of new or improved prior CADe systems • Since the necessary performance level of CADe will be known from STAGE ONE, now it will only be necessary to demonstrate that the system’s STAND ALONE performance meets or exceeds the earlier-specified stand alone performance standard of sensitivity and FPs/image. • Similar to the testing of image acquisition devices in terms of physical image quality metrics (e..g, MTF, SNR, etc.) • Test result could include, for various “populations”: sensitivity and FPs/image, and perhaps FROC - Note location is included

9. Suggest a Two-Stage Method of Evaluation • STAGE TWO (b): • An independent “technology assessment institute” (TAI) could be tasked with the assessment of the stand alone performance of all new or improved CAD devices • Institute would have a protected and sufficiently large database with appropriate distributions of cancer/abnormality sub-types and prevalence to allow for random sampling of cases for a CAD system assessment • Can also vary the “enrichment” of different case types • Integrity of the “tests set” would be maintained since each test set would be randomly drawn from a LARGE set (within the TAI) according to a specific case distribution and cancer prevalence

10. Suggest a Two-Stage Method of Evaluation • STAGE TWO (c): • An independent “technology assessment institute” (TAI) could be tasked with the assessment of the stand alone performance of all new or improved CAD devices • Only the test result (sensitivity and FP/image) would be given back to the manufacturer, so the manufacturer would not be able to “train to the test set” • Evaluation method eliminates the variation of the radiologist mindset, skill, level of training, etc. from the evaluation • Retesting of the CADe system by the TAI would yield measures of reproducibility (since a different testing set would be randomly drawn from the main TAI database) • Blackbox of the manufacturer is maintained • Output from such a TAI evaluation would be “acceptable” for FDA submissions

11. Technology Assessment Institute(TAI) • Independent • Supported by government grants/contracts and industry (fee for testing service) • Could have oversite from a scientific association such as AAPM (American Association of Physicists in Medicine) • Expect would be used for FDA 510(k) for all types of CADe including mammography [SF & FFDM] in order to test the “label” and include the test score in the “label”

12. Summary of Proposed Evaluation Method - 1How to evaluate a new CADe system??? TAI approach allows for: • Least burdensome approach to demonstrate substantial equivalence • Standardization of testing including scoring method and ground truth • Maintenance of the integrity of “test set”, and avoid reuse of data for testing -- Eliminates teaching to the test concern • Case mix: Allow for enrichment of a test set with different lesion types, cancer prevalence, etc. to match the clinical context • Includes random samples of population • Reproducibility metrics

13. Summary of Proposed Evaluation Method - 2How to evaluate a new CADe system??? TAI approach allows for: • Reader Mix: Realization that in ultimate clinical practice, the user will change with different CADe systems - so potentially could take user out of the evaluation • TAI approach eliminates reader variability, bias, training level, performance level, problem with small number of readers that may not generalize, etc. • Clinical Utility would have been shown in Stage ONE

14. Summary of Proposed Evaluation Method - 3How to evaluate a new CADe system??? - TAI • However, need to consider role of user/computer interface of any particular CADe system • Display device considerations • Necessary for only large differences in prompting methods • Need to consider situation in which different types of FPs occur • Could be realized during post market evaluations • Note that the TAI test score would be included in “label” • [Recall that for CADe in mammography the output helps decide if additional imaging is needed, it does not inform on any other patient management decisions.]

15. Consider the role of a TAI in pre- and post- market evaluations for CADe.Will also be useful for CADx.Thank you