Interalgorithm Study using CT Images of synthetic nodules……….

3. Interalgorithm Study using CT Images of synthetic nodules……….

4. Objectives of the QIBA 3A Group • An inter-algorithm study, in the same way QIBA has been working on inter-reader,inter-scanner, and inter-site.  We will also connect it to the analysis section of QIBA Profile.  • The aim of the study is to estimate inter- and intra-algorithm variability by the volume estimation of synthetic nodules from CT scans of an anthropomorphic phantom (according to the work of the QIBA 1A Group (see Dr. Petrick‘s paper , SPIE 2011) • For the impelmentation of the obejctives a challenge could be organized (?)

5. Study Motivation Motivation for the Study Study Design Dataset Algorithms Analysis Protocol Analysis method Result evaluation Result presentation Result evaluation for the QIBA protocol

6. Motivation

7. Motivation • Changes in nodules volume is important for diagnosis, therapy planning, therapy response evaluation • Measuring volume changes requires high accuracy in measurement of absolute volume • Ground true has to be exact measured. This is not the case by data annotation (inter- and intra-observer variability) • Volumes of synthetic nodules are measured (high accuracy) • Therefore it make sense to use such data (as ground truth) in order to calculate accuracy measurement of algorithms • The study results could be combined with the QIBA 1A Group work. This combination will improve the QIBA volumetric CT Profile development. • The study has to be complementary to the study of the 1A Group: • 1A study: Radiologists and synthetic nodules (inter- intra- observer variability) • 3A Study: Algorithms and synthetic nodules: intra-algorithms variability) • Using of the same technological basis (scanner type, Philips (1A Group data, synthetic nodules)) to secure independence for inter-scanner variability (except BIAS between different serial numbers)

8. Analysis procedure (analysis Protocol according Bio-change or Volcano Challenge?) • Describe the overall analysis procedure (use as example the Bio-change Challenge Protocolhttp://www.nist.gov/itl/iad/dmg/biochangechallenge.cfm: How to Participate in Biochange Challenge (Dr Fenimore) Download and read the Bio-change Challenge Protocol and email or fax Statement of Interest to NIST. Download the Bio-changeChallengeSeries from the NIST FTP site and from the NBIA RIDER collection as described in the Protocol. Run your change analysis algorithm or CAD tool in your lab on the validation data. Report your change results in one of the required formats and send a Participation Agreement signed by the your team leader to NIST by January 18, 2011. NIST will analyze the reported results, comparing them to the limited available ground truth as described in the Protocol. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without publicly identifying individual scores by Participant.

9. Has to be discussed during the meeting March 17th 2011 OUR CONTEXT is QIBA • The aim of the study is not a challenge to know: • Who is the best image analysis algorithm • The aim of the study is to gain knowledge for the QIBA profile (paragraph 9 concerns image analysis) • Avoid competition • Support Cooperation, conjoint approach

10. Data/Nodules/Algorithms • Phantom data as used for the 1A Group Study (see Dr. Patrick’s publication, SPIE 2011) • 1C clinical data (Dr Fenimore/ Phantom –synthetic data) • Training dataset • Test dataset • Or only test dataset (analyze without trainings dataset) data pool specification (simple geometries, up to anatomical representations) (if clear than we could propose an experimental design) • Nodule classification concerning position. Shape and margins of the nodule … (according to the QIBA Profile) • Algorithm classification • Analysis protocol: • each algorithm is applied to the dataset separately • each algorithms is applied to the “training” data set • each algorithms is applied to the “test” data set • Statistical analysis of the analysis results of each algorithm separately

11. Algorithm vendors and Algorithm classification examples from the literature • Academia and non profit organizations • Industrial vendors (for example possible vendors, according to the Volcano 2009 challenge could be: Siemens, Phipps, MeVis, Kitware, Definiens, VIA CAD etc… • Description/Classification of the algorithms: according to the grade of user intervention is needed (for example Volcano’09, A. P. Reeves et al): • Totally automatic using seed points • Limited parameter adjustment (on less than 15% of the cases) • Moderate parameter adjustment (on less than 50% of the cases) • Extensive parameter adjustment (more than 50% of the cases) • Limited image/boundary modification (on less than 15% of the cases) • Moderate image/boundary modification (on less than 50% of the cases) • Extensive image/boundary modification (more than 15% of the cases) Or: • M. Gavrielide et al., Noncalcified Lung Nodules, Volumetric assessment with thoracic CT (RSNA 2009): • Semi automated • Manually derived Semi automated algorithms are typically initiated by defining a region of interest around a nodule or by a user-provided point inside the nodule area. Depending on the application, segmentation algorithms are then employed to delineate nodules from the surrounding lung parenchyma and neighboring structures such as attached vasculature and pleural surfaces. Manually derived methods require users to interactively delineate nodule boundaries, typically in a section-by-section fashion; this is followed by an application of three-dimensional software to merge the two-dimensional boundaries into a volume. The estimate of nodule volume is then based on the total number of voxels within the segmented region. The majority of volume measurement methods use voxel counting

12. IEEE Transactions on medical imaging, vol. 28 MICCAI Algorithm Classification according 3D Segmentation in the Clinic: A Grand Challenge II - Liver Tumor Segmentation, Xiang Deng and Guangwei Du, Center for Medical Imaging Validation, Siemens Corporate Technology (citation) Algorithm Classification: • An automatic segmentation algorithm does not require any user intervention. • A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation. • An interactive algorithm requires manual editing of the final results. Abstract: In this paper, we present the organization of a competition of liver tumor segmentation techniques. The liver tumor segmentation competition is part of the workshop "3D Segmentation in the Clinic: A Grand Challenge II" at Medical Image Computing and Computer Assisted Intervention 2008 conference. The goal of this contest is to compare the he performance of different algorithms for segmenting liver tumor from contrast enhanced CT images. Several organizing topics are described, contrast including the motivation for organizing this competition, training and testing dataset evaluation methods

13. MICCAI Liver Challenge proposed metrics (citation)

15. Lucila Ohno-Machado An introduction to calibration and discrimination methods, HST951 Medical Decision Support Harvard Medical School, Massachusetts Institute of Technology • Sensitivity Sens = TP/TP+FN • Specificity Spec = TN/TN+FP • Accuracy = (TN +TP)/all

16. Other • Box plot (for example: for each data set, nodule volume vs. method, relative Volume) Volrel= 100%* (Volest – Voltrue)/ Voltrue Volrel= 100%* (Volalg – Volsynthetic nodule)/ Volsynthetic nodule We propose: A simple but well defined and reliable approach

17. We propose: keep it simple • Algorithm Classification: • An automatic segmentation algorithm does not require any user intervention. • A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation. • An interactive algorithm requires manual editing of the final results.

18. Algorithm Evaluation Method • Known: Vsn • Measurement: Valg • StdvValg (1) <- same Phantom, multiple measurements, same algorithm, multiple seed points (two groups of algorithms: semiautomatic, manually) • StdvValg (2)<- different phantoms (according QIBA profile: nodule classification), same algorithm: • StdValg (nodule class) • StdValg(all nodule classes) • Bias out of digitalization in the area of lesion margins (Voxels/Pixels on the lesion margins, mixed pixels) (Group 1A, Group 1C, Group 1B, QIBA VIA Profile, Definitions and how should be calculated?) • Achievements: • Transparent and Objective comparability between different algorithms • One independent Error value for further investigations of clinical data

19. Definition of the Study Data/Measurements/Metrics / • Which data: • 1A (Phantom/synthetic data) (defined experimental conditions, experimental design Phantom Data for dV?) • 1B Clinical data according to that: do we have we to separate the study design as following? • Study design for Phantom/Synthetic data (focus) • Study design for Clinical data • Metrics

20. Decision

21. Result information for the participants • Each participant has to be informed (only their own algorithm results) • Publication of the results (all participants) • Using the results for the QIBA protocol: knowledge exploitation for the QIBA protocol • Resulting recommendations for FDA (Is this possible?)