QIBA CT Volumetrics Group 1B: (Patient Image Datasets)

1. QIBA CT Volumetrics Group 1B: (Patient Image Datasets)

2. Charge • To agree on questions to be answered by these Reference Datasets • To identify requirements for those datasets, based on questions to be answered • To identify existing datasets that can be leveraged to provide desired datasets

3. Questions (overview) • What level of accuracy and precision can be achieved in measuring tumor volumes in patient datasets? • What level of reproducibility in estimating change can be achieved when measuring tumors in phantom datasets? • What is the minimum detectable level of change that can be achieved when measuring tumors in patient datasets under a “No Change” condition? • What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? • What is the effect of slice thickness on estimating change in tumors using patient datasets?

4. Initially Agreed to First Pursue: • What level of accuracy and precision can be achieved in measuring tumor volumes in patient datasets? • What level of reproducibility in estimating change can be achieved when measuring tumors in phantom datasets? • What is the minimum detectable level of change that can be achieved when measuring tumors in patient datasets under a “No Change” condition? • What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? • What is the effect of slice thickness on estimating change in tumors using patient datasets?

5. Next Steps • Define requirements and Design Experiments for these two questions

6. Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes • Specific Aims • Investigate both bias and variance of both readers and algorithm-assisted readers in measuring volumes, diameters and bi-directional diameters of lesions • Investigate inter-observer variability in each task • Investigate Intra-observer variability in each task ??? (NOTE: here observer should be interpreted broadly – as reader measuring manually for diameters as well as algorithm-assisted reader measuring contours).

7. Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes • METHODS and MATERIALS • LIDC dataset • lesions with “known size” will be used (size based on contours from 4 LIDC readers) • Each lesion will have boundaries • Only a single time point is needed (no followup needed, no diagnosis needed) • Need to identify which nodules to use – criteria?

8. Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes • METHODS and MATERIALS • Like NIST Biochange 2008, lesions are identified and coordinates provided to readers. • Reader tasks • Manually mark 2 Diameters (w/o LIDC marks) • Longest diameter and perpendicular diameter • Also semiautomated contour of lesion • From contour, determine volume, longest diameter and diameter perpendicular to longest diameter • Have readers perform some cases > 1 (intra reader variation)

9. Data Collected

10. Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes • METHODS and MATERIALS • Analyses • LIDC would be considered “truth” • Estimate bias of each reader • Volume • Diam (manual and Assisted) • Product of diameters (LD x PD) (Manual and assisted) • Estimate inter-reader variability • Reader vs. “gold standard” • Reader vs. reader • Intra-reader variability

11. Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes • Questions • How many cases? • How many readers? • Case composition (all spherical? Some spiculated? Range of sizes?) • Enough of each subgroup to perform a stat. sign analysis? • (reader bias on spherical nodules is XX; • (reader bias on spiculated nodules is YY) • Do we want to do both size and shape subgroup analyses?

12. Questions

13. Question 2 – What is the min detectable level of change in patient datasets under a “No Change” condition? • Specific Aims • For patient datasets acquired over a very short time interval (presumably the “no change” condition”) investigate variance of both readers and algorithm-assisted readers in measuring change in volume, diameter and bi-directional diameters of lesions (here, the expected value of the change should be zero) • Investigate several change metrics such as: • Absolute value of change • fractional change in volume/diameter • Categorical variables (progression, regression, etc.)?? • Investigate inter-observer variability in each task • Investigate Intra-observer variability in each task ??? (NOTE: again, observer should be interpreted broadly – as reader measuring manually for diameters as well as algorithm-assisted reader measuring contours).

14. Existing Resources • RIDER – MSK Coffee Break Experiment (No Change Condition) • 32 NSCLC patients • Imaged twice on the same scanner w/in 15 minutes • Thin section (1.25 mm) images • Manual linear measurements performed by 3 readers; volume obtained from algorithm.

15. Questions • METHODS and MATERIALS • MSK Coffee break experiment patient datasets • Repeat scan of lesions over short time interval --- “No change” condition • True size is unknown

16. Questions • METHODS and MATERIALS • Like NIST Biochange 2008, lesions are identified and coordinates provided to readers. • Reader tasks • For each lesion, manually mark 2 Diameters • Longest diameter and perpendicular diameter • Also each reader obtains a semiautomated contour of entire volume of lesion • From contour, determine volume, longest diameter and diameter perpendicular to longest diameter • Keep actual contour coordinates/mask (TBD) • Randomize order of cases so reader does not simultaneously review both scans of a patient (does not see both scans of same lesion); has to contour them independently.

17. Data Collected

18. Question 1 – Accuracy and Precision (aka Bias and Variance) in measuring tumor volumes • METHODS and MATERIALS • Analyses • Estimate variance of different change metrics for • Volume • Diam (manual and Assisted) • Product of diameters (LD x PD) (Manual and assisted) • Estimate inter-reader variability • Intra-reader variability ?? (can we do that with data we would have? Or would we have to specifically introduce the exact same cases multiple times?)

19. Questions to be Pursued in a Future Phase

20. Questions 2. What level of reproducibility in estimating change can be achieved in measuring tumors in phantom datasets? • RECIST change vs. volume change • Investigate just variance (not bias)? • inter-observer variability • Intra-observer variability • Change metric – absolute value? fractional change in volume/diameter? categorical variable? For this question: • Variety of lesions with known size • Compare “size” of different lesions somehow (TBD) • Use different existing lesions and treat them as though they were the same lesion at different time points. • Physically alter lesions over time and scan at both time points (Bob Ford’s water balloon experiment)

21. Questions 4. What level of reproducibility in estimating change can be achieved in measuring tumors in patient datasets with “Unknown Change” condition? • RECIST change vs. volume change • Investigate just variance? • inter-observer variability • Intra-observer variability (would be good to have more than 2 time points; may not exist in RIDER). • Change metric – absolute value? fractional change in volume/diameter? categorical variable? • Look at effects of lesion size for a specific (thin) slice thickness For this question: • Variety of lesions (true size may be unknown) • Patient datasets with same lesions at different time points. • Lesions may or may not have changed size – change is unknown. • RIDER datasets – unannotated as of yet. We have identified about 20 lesions of various sizes. Possible cases from RadPharm.

22. Questions 5. What is the effect of slice thickness on estimating change in tumors using patient datasets? • RECIST change vs. volume change • Investigate just variance? • inter-observer variability • Intra-observer variability • Change metric – absolute value? fractional change in volume/diameter? categorical variable? • Look at interactions between slice thickness and lesion size (and other lesion characteristics such as shape, margin, etc.) For this question: • Also Variety of lesions (true size may be unknown) • Patient datasets with same lesions at different time points, all done originally with thin slices; create a thick slice series (average together adjacent images). • Lesions may or may not have changed size – change is unknown. • Estimate change on thin and thick series and compare • Thin Slice RIDER datasets that can be fused together. Other cases from RadPharm?