Clinical Description – Part I William Sacks, PhD, MD--ODE

R2 Technology, Inc. ImageChecker CT CAD Software SystemP030012 Radiological Devices Advisory Panel February 3, 2004 Clinical Description – Part I William Sacks, PhD, MD--ODE Sacks

Outline • Character of device • Clinical utility • Instructions for use • Issues new to this device Sacks

Character of device • For chest CT scans • For CTs done for any indication • Detects solid lung nodules • E.g., not ground glass opacities (GGOs) • Nodules between 4-30 mm • Computer-Aided Detection (CAD) Sacks

Computer-Aided Detection(CAD)/Diagnosis(CADx) CAD CADx Sacks

Clinical Utility • Many nodules missed in clinical practice: • other pathology distracts • hundreds of images per exam • CAD intended to reduce missed nodules Sacks

Instructions for Use • Reader reviews films unaided first • CAD marks candidate nodules • Reader looks again in vicinity of marks • If CAD fails to mark a nodule judged actionable on unaided review, reader should retain initial judgment Sacks

Issues new to this device • CAD target • Definition of “truth” • Unit of analysis • Endpoints Sacks

CAD TARGET • Not malignant nodule • but • Actionable nodule Sacks

DEFINITION OF “TRUTH” • Not biopsy (tissue histology) • but • Expert panel Sacks

UNIT OF ANALYSIS • Statistical unit = person • further broken down into • Lung quadrant Sacks

ENDPOINTS • Sensitivity and specificity (or FPR) of action recommendation • and/or MRMC ROC Sacks

CLINICAL STUDY • 3 expert radiologists on panel to determine “truth” for each nodule • 15 other radiologists with range of experience, called “the readers” • 90 subjects (360 lung quadrants) • 15 readers used 100-point scale for confidence-in-actionability of each case Sacks

CLINICAL STUDY • Presentation by • Nicholas Petrick, Ph.D. Sacks

R2 Technology, Inc. ImageChecker CT CAD Software SystemP030012 Radiological Devices Advisory Panel February 3, 2004 Clinical Conclusions – Part II William Sacks, PhD, MD--ODE Sacks

What does gain in Azof 0.02 (95% CI: 0.01, 0.04)mean clinically? Sacks

When CAD is used according to instruction to retain all judgments of actionability even if unmarked by CAD... Sacks

... user always maintains or increases sensitivity, and always maintains or increases FPR, as well. Sacks

Gain of 0.2 in AZ understates relative gain in sensitivity. Sacks

ROC 1 ΔFPR Aided FNRU ΔSe =ΔFNR Unaided SeU Chance line (guessing) TPR 0 1 0 FPR Sacks

ROC 1 Loss of Se only possible if instruction not followed Aided Realistic Unaided Chance line (guessing) TPR 0 1 0 FPR Sacks

So any stat sig improvement in Az means an even greater relative gain in Se, without “undue” increase in FPR. Sacks

Can we infer from improved user performance in clinical study ... Sacks

... that user will improve performance with CAD in clinical practice? Sacks

For example, in actual clinical practice, the unaided Az could be lowered by failure to read first as one would normally read (i.e., with adequate vigilance). Sacks

If this were to happen, then the aided Az could also be lower than current practice (in the complete absence of a CAD). Sacks

Two rules, if followed by user, will prevent that from happening: • Always read unaided first, and as carefully as if you had no CAD. • Never back off from unaided judgment of actionability of a nodule if CAD fails to mark it. Sacks

Summary • Data show stat sig improvement of Az of aided over unaided readings with CAD • If “Always” rule followed then aided curve will be higher than current practice • If “Never” rule followed, then gain in Az implies even greater relative improvement in Se of aided readings over current practice, without “undue” increase in FPR Sacks

Clinical Description – Part I William Sacks, PhD, MD--ODE