Agenda : Certification Platform: performance for training - I round - II round

1. XIV PMT meeting – Dec 5, 2012 • Agenda: • Certification Platform: performance for training - I round • - II round • Feedback from naives • Beta testers proposals • Congress presentations • Paper “check for axes”

2. Platform: Training Phase 2 Images (Scheltens=0,3) Feedback Certification set Feedback + - 6 Images (Scheltens=0,1,2,3,4,4) Certification set 10 Images (2 x Scheltens=0,1,2,3,4) Feedback + -

3. Platform: Training round I

4. Platform: Training rounds I and II Tracers received a verbal feedback in order to minimize misunderstanding possibly originated by the incomplete functionality of the platform at this stage, or due to the underlying statistical operations for the visual feedback. No verbal feedback for Round III

5. Platform: Training round I A qualitative check (QC) was carried out slice by slice, same procedure as for the benchmark images, but with the help of the platform: 1) Platform usually reflects properly the correctness of segmentation. Limitations lie in CSF pools (a.: correct segmentation is not properly rendered with “green”), and in the minimum variability allowed to the naïve where masters were in great agreement with each other (b.) a. b.

6. Platform Training round I: Performance * Minor lacks of compliance with HP; § Major lacks of compliance with HP; §§§ More relevant segmentation mistakes than just compliance to HP

7. Platform Training round II: Performance Very atrophic sbj (Scheltens=4). Better distribute Sch=4 across training rounds next time?

8. Training Platform Tracer Area (mm^2) Area delimited by the tracer contour for the current slice. Master Min Area (mm^2) , Master Mean Area (mm^2) , Master Max Area (mm^2) Area delimited by the master minimum/mean/maximum contour for the current slice. Distance Integral: Line integral of the distance ratio (see definition above for “distance ratio”) values along the tracer contour, normalized by the contour’s number of point, for the current slice. Hausdorff Distance (mm): Hausdorff distance (see definition above) between the tracer contour and the master mean contour for the current slice. Dice Index w.r.t. Min, Dice Index w.r.t. Mean, Dice Index w.r.t. Max Dice similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice. Jaccard Index w.r.t. Min, Jaccard Index w.r.t. Mean, Jaccard Index w.r.t. Max Jaccard similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice. True Positive Area w.r.t. Min (mm^2), True Positive Area w.r.t. Mean (mm^2), True Positive Area w.r.t. Max (mm^2) Tracer true positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice. False Negative Area w.r.t. Min (mm^2), False Negative Area w.r.t. Mean (mm^2), False Negative Area w.r.t. Max (mm^2) Tracer false negative (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice. False Positive Area w.r.t. Min (mm^2), False Positive Area w.r.t. Mean (mm^2), False Positive Area w.r.t. Max (mm^2) Tracer false positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

9. Certification Criteria (to be defined) Hypothesis so far (generated versus quality check) is that: Jaccard < 70 denotes major problems in hippocampal segmentation Jaccard < 80 denotes incomplete compliance with the HP criteria A hypothesis might be that at least Dice ≥ 90 and Jaccard ≥ 80 be required for qualifying. Anyway we first evaluate performance at third round (Wide statistical analysis may be required for adequate criteria selection…)

10. Feedback from tracers Would find it easier to follow a video, replacing in person communication (video, and also inter-linked pop-up mesg -maybe for the first two training images?- OK but later) For now, legends to figures and table of contents are being added to the HP to facilitate reading HP edited based on appropriate feedback from tracers, and re-sent to tracers HP will be resent to panelists after we received comments from all naïves Visualization improvements (are being implemented already)

11. b-testing Masami Nishikawa: Asks: access to protocol for validating VSRAD (common algorithm for hippo segmentation in Japan) Offers: Further validation branch for the SOPs project, on E-ADNI human phantoms (verbal offer by Matzuda of repeating segmentations on J-ADNI MRIs, having over 300 hippos segmented based on local protocol already)

12. Congress Presentations • CTAD (Montecarlo, October 29-31 2012) • Definition of the Harmonized Protocol for Hippocampal Segmentation • DGPPN Symposium (Berlin, November 21-22 2012) • (Andreas Fellgiebel, Stefan Teipel) • Segmentation of the hippocampus: Towards a joint EADC-ADNI harmonized protocol • Biomarkers for Brain Disorders (Cambridge, February 3-5 2013) • Definition of harmonized protocol for hippocampal segmentation • AD/PD (Florence, March 6-10 2013) • Definition of Harmonized Protocol for Hippocampal Segmentation • AAN (San Diego, March 16-23 2013) • EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation

13. Papers describing the project Survey of protocols (preliminary phase; Published, JAD 2011) Operationalization (preliminary phase; Accepted, Alzheimer’s & Dementia, MS n. ADJ-D-12-00094) Axes check short report (Brescia Team, to be shared with coauthors) Delphi consensus (Brescia Team, in progress) Master tracers’ practice and reliability (Brescia Team, in progr) Development of certification platform (Duchesne and coll) Validation data (Brescia Team – companion paper 1) Protocol definition (Brescia Team – companion paper 2) Validation vs pathology (TBD) DONEIN PROGRESSPLANNED

14. GOLDSTANDARD VALIDATION VS CURRENT PROTOCOLS ASSESSMENT OF SOURCES OF VARIANCE TRAINING SET DEVELOPMENT VALIDATION VS PATHOLOGY 20 naïve tracers 5 master tracers 1 tracer Local Protocol ADNI scans: 2 x 5 Scheltens’s atrophy scores x 2 sides x 2 magnet strengths (1.5-3T) Training ADNI scans: 10 at 1.5T x 2 sides x 7 SUs x 2 tracing rounds Total per rater: 40 hippos QUALIFICATION QUALIFICATION Harmonized Protocol ADNI scans: 2 x 5 Scheltens’s atrophy score x 2 sides x 2 magnet strengths (1.5-3T) Total per rater: 40 hippos Harmonized Protocol ADNI scans: 2 x 5 Scheltens’s atrophy score x 2 sides x 2 magnet strength (1.5-3T) Total per rater: 40 hippos Harmonized Protocol: Pathological datasets: Mayo Clinic and NYU Total: about 40 hippos Assessment of variance due to rater and center REFERENCE PROBABILISTIC MASKS with 95% C.I. Best 5 naïve tracers Assessment of agreement with volume on pathology or ex vivo MRI and correlation with neuronal density Harmonized Protocol ADNI scans: 2 sides x 5 Scheltens’s atrophy scores x 3 time points (bl-1y-2y) x 3 scanners (+ retracing @ bl) x 2 magnet strengths (1.5-3T) Total per rater: 240 hippos Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater TRAINING SET

15. GANTT