Preclinical-Bench Testing II Using Human Observers to Objectively Measure and Evaluate Imaging Performance of Digital

1. Preclinical-Bench Testing IIUsing Human Observers to Objectively Measure and Evaluate Imaging Performance of Digital WSI Systems Max Robinowitz, MD FDA Center for Devices and Radiological Health October 22, 2009

2. Object Related Diagnosis Please look at this picture Identify all that you see What do you think they represent? What is your diagnosis?

3. From Kayser K et al Diagnostic Pathology 2006

4. How Can We Objectively Evaluate If Image Quality Presented to Pathologists is Fit-for-Purpose? How can we quantify the contribution of the image qualityto a final tissue-based diagnosis? How can we quantify contribution added from the eye &brain of the pathologist? How can we quantify the ability of pathologists to search & detect different objects rendered by OM & WSI? How can we identify the contributions of the ergonomics and human factors of the different work stations?

5. Role of Preclinical-Bench Testing Physical & biological phantoms may provide objective data for prediction of diagnostic performance of pathologists with the optical microscope & different WSI systems in the real world in a manner similar to x-ray phantoms The general application of WSI to routine surgical pathology practice potentially involves allof clinical medicine All clinically essential areas within the whole spectrum of surgical pathology may have to be tested

6. Human Observer Studies with Biologic & Human Histologic Specimens Phantoms • Minimize the subjectivity of study pathologists by emphasizing each pathologist’s assessments of image quality and information contentbefore the overlay of high-level intellectual diagnostic interpretation • Targets can be constructed from biological materials of known size, number, fine morphological features & spatial distributions to quantify image quality---objectively

7. Controlling Subjectivity in Performance of Real World Tasks of Pathologists • Pre-test tutorial to standardizecriteria for the study pathologists. • Pre-specify the features to observe, enumerate, using criteria thatminimize requirements for higher-level interpretation by pathologists • Use biological objects in study as the equivalent of a Snell eye chart to test eye glasses. • Task: Recognition of shapes & sizes in images using description rather than diagnostic terms --- Have pathologist quantify the detailed features of the image

8. Diatoms = Algae with Known Size & 3D Textural Features Requiring 20 & 40X Objectives to View Details in Image 97 Diatoms ranging from about 2 to 100 micrometers in diameter. Note: fine 3D internal features that can be objectivelymeasured &enumerated From Kemp K diatoms.uk.com

9. Objective “Open Book Exam” Study with Human Specimens • Controlled, non-subjective evaluation of image quality rendered by OM versus the image quality rendered by WSI systems • Reproducible, optimal preparations of normal and abnormal human specimens on TMA cores and whole slide

10. Critical objects & Features to evaluate • Accuracy & precision of pathologists for identification of objects and features in targets • Clinically significant objects, e.g., cancer microinvasion, nuclear chromatin structure, nucleoli, cytoplasmic vacuoles, inclusions, granules & striations, architectural growth patterns, etc.

11. Critical Variables to evaluate • Effects of tissue preparation, slide thickness, different stains, different adjustments & settings of OM and WSI system • Variability inplanes of focus selected by WSI on different runs • Variability in pathologists with different types of image features

12. Tasks for Study Pathologists That CanUse Cores or Whole Slide Specimens • Task 1 Search & find/detect objects and textures to test image quality, focus and fidelity, dynamic range, color fidelity across whole slide • Task 2 Enumerate number & measure objects • Task 3 Classify detected objects

13. Task 4 for Study Pathologists is Limited to Whole Slide Specimens • Task 4 Diagnose based on the enumeration and cumulative semi-quantitative scores from the objects and features from larger areas that exceed the size of TMA cores (mimics intended use) • Quantify with Scoring system that is a composite of multiple simple features—some require the optical resolution at 40X & z-axis focusing

14. Tissue Microarrays with CoresofNormal & Abnormal Features Selected by Experts Individual core 9X9 grid TMA Paraffin block TMA cores from ROIs of multiplespecimens, in precise order of a grid on single slide, allows efficient viewing by study pathologists for only some of the fit-for-purpose testing

15. Tissue Microarray (TMA) Tissue CoresDiameter Sizes and Numbers/Slide • Core sizes are 0.6, 1.0, 1.5 and 2.0  mm most common. Most popular is 1.0 mm. •  0.6 mm = standard 40X objective field of view • 1.0 mm = standard 20X objective field of view •   Larger cores can be made but even 1.5 and 2.0 are very hard to work with, but are good for normal tissues where you want to show a number of features. • 0.6 = 500 cores/slide; 1.0 = 200 cores/slide • 1.5 = 75 cores/slide; 2.0 = 35 cores/slide

16. Tissue Microarray (TMA) Tissue CoresApplications and Limitations • Tissue microarrays are suitable to test pathologists’ ability for recognitionofcellularobjects (features) and some limited spatial tissue growth patterns • TMAs are NOTsuitable for testing the tasks of search & find/detection of ROI within the multiple different diagnostic areas on a whole slide specimen as required in the real-world of surgical pathology

17. Bench-Testing for Realistic Diagnostic Tasks: Combinations of Multiple Features • Pre-test tutorial with schematic with the features to be identified and compared with slide • Slide with multiple conditionstosearch, detect & describe according to their location on the whole slide and its image on WSI, e.g. upper right quadrant, lower right quadrant • Match & Describe each abnormality according to the schematics and pre-test “standardized terminology” III IV DCIS

18. Tubule Formation Majority of tumor (>75%) 1 Moderate degree (10-75%) 2 Little or none (<10%) 3 Mitotic Count 0-9 mitoses/10 hpf 1 10 -19 mitoses/10 hpf 2 20 or > mitoses/10 hpf 3 Nuclear Pleomorphism Small regular uniform cells 1 Moderate nuclear size and variation 2 Marked nuclear variation 3 Combined Histologic Grade Low grade (I) 3-5 Intermediate grade (II) 6-7 High grade (III) 8-9 Nottingham BrCa (Ordinal) ScoreControlled Task for OM & WSI This is just one of many clinically relevant examples

19. Heirarchy for Grading Dx Errors10 levels of increasing complexity ofdiagnosticinterpretation & final diagnosis nomenclature Image Based High-Order Intellectual Overlays Mireskandari M, Kayser G, Hufnagl P, Schrader T, Kayser K Diagnostic Pathology 2004

20. FDA Desires Panel Input • What roles can bench-testingof OM and WSI serve for guiding the designs for clinical studies in real world surgical pathology practices? • What safety and effectiveness concerns can be resolved with pre-clinical studies using selected histopathology specimens, such as TMAs and whole slides? What are the limitations? • What concerns require clinical trials with representatives of theintended pathologists who must examine whole slides in the real worldsetting of surgical pathology practice?