Agenda : Hippocampal Label Expansion project Certification Platform: performance/criteria

1. XV PMT meeting – Jan 30, 2013 • Agenda: • Hippocampal Label Expansion project • Certification Platform: performance/criteria • Questionnaire • Congress presentations • Papers

2. Label Expansion - Design 4 Subjects = 135 Hippocampi = 270

3. Hippo Label Expansion USD 95239 AA (50%), Bioclinica, Brain Image Analysis, Ixico, Roche, Synarc, True Positive Medical Devices (Louis Collins, Jan 29) (50%) Laval: OK managing labels for qualification of Algorithms Fund from Can AA will better enable that, and completion of current platform/certification criteria for human tracers (Wide logistical analysis for certification criteria can be carried out by Abderazzak at Laval) Deadline: 3 months from $ receipt Tracers: 1) Masami (Jaccard=83 in the whole Certification phase) 60 images (overlap with his beta tester project for Jap VSRAD algorithm) 2) Martina 19 images Contacted: Rossana (Master; 28 img?), Dominik (Master; 28 img?)

4. Publication policy AVAILIABILITY OF DELIVERABLES: Needs being changed? NO for the qualification labels (they should not be delivered anyway) Maybe for the labels from the validation. Anyway, SD notes that these labels are FEW, and from few subjects: developers will have all what they need from the Expansion project,  no need to change policy on availability of labels

5. Publication policy AVAILIABILITY OF DELIVERABLES The Harmonized Protocol (i.e. the text description and illustrative material), the digital master tracer masks, and digital training sets (masks developed by the 5 best naïve tracers) will be available only to beta-test users in the period (about 1 year) between the end of the Delphi Panel and the publication of the full validation resultsOr ADFEQUATE QC (i.e. the analysis of the whole set of hippocampal tracings by all naïve tracers). To qualify as beta-testers, users will submit proposals to the SC; if accepted, beta-testers will be granted access to deliverables under conditions dictated by a written cooperation agreement that will be signed by the user and the Chair of the SC. Assuming mutual benefit, the cooperation agreement will include provisions related to scientific publications, donations, and feedback that beta users will provide on their exploitation of the Harmonized Protocol.

6. ADNI Publication policy AVAILIABILITY OF LABELS ON THE “SODIUM” WEBSITE This would soon be an issue for the Expansion Expanded labels may be back-transformed to native space: users will only need the ADNI code

7. Platform: Training Phase TRAINING 2 Images (Scheltens=0,3) Feedback Feedback 6 Images (Scheltens=0,1,2,3,4,4) CERTIFICATION 10 Images (2 x Scheltens=0,1,2,3,4) Feedback VALIDATION

8. Training Performance & Validation

9. Emma, round 1 (1.5T), Jaccard: 0.69 left and 0.65 right

10. Yawu, round 1 (1.5T), tot Jaccard: 0.71 does not distinguish amy from hippo; includes fornix

11. Melanie, img: bq_f_15T, total Jaccard: 0.76 She does not separate properly the indusium griseum, and sometimes includes fornix. These errors are repeated in more slices where the criterion applies. All the rest is ok or really minor.

12. Masami, img: bq_a_3T, total Jaccard: 0.82 Minor mistake in applying the HP criteria, or small inconsistencies in potentially ambiguous boundaries These are all of his divergence points. We can not really say that he is “not compliant” with the HP

13. Michel, subject bq_F_3T (Qualific) Jaccard: left 0.873, right 0.870

14. Jaccard indices in Qualification

15. Reliability in III round

16. Certification Criteria (to be defined) Hypothesis so far (generated versus quality check) is that: Jaccard < 70 denotes major problems in hippocampal segmentation Jaccard < 80 denotes incomplete compliance with the HP criteria for normal subjects (may be admissible for atrophic subjects: Enrica, bq_r_15T, J=0.724, tracings ok) Combination of values needed (Wide statistical analysis needed. See slide 3, Abderazzak at Laval and Can AA funds for Laval)

17. Questionnaire How long does it take to you to segment one hippo based on the HP? Do you believe that your comprehension of the hippocampus and your way to segment has improved after the training based on HP? Which are the parts of the HP that were less clear to you? Were there regions in the Hippocampus where you did not know how to segment due to lack of instruction for that specific point or case? (if yes, please attach figures illustrating them) Questions regarding the platform as required by SD We can use SD’s Survey Monkey’s account

18. Congress Presentations • - Biomarkers for Brain Disorders (Cambridge, February 3-5, 2013) • Definition of harmonized protocol for hippocampal segmentation • AD/PD (Florence, March 6-10, 2013) • Definition of Harmonized Protocol for Hippocampal Segmentation • AAN (San Diego, March 16-23, 2013) • EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation • IEEE Medical Measurement and Applications conference (Gatineau –Quebec-, May 4-5, 2013. • Platform for Training and Qualification • AIC-AAIC (Boston, July 13-18, 2013) • Training of Naïve Tracers; Statistics for Certification Platform • (SINDEM (Ita Soc for Neurology of Dementias) Perugia March 13-15 Definition of harmonized protocol for hippocampal segmentation

19. Papers describing the project Survey of protocols (preliminary phase; Published, JAD 2011) Operationalization (preliminary phase; Accepted, Alzheimer’s & Dementia, MS n. ADJ-D-12-00094) Axes check short report (Brescia Team, ADJ-D-13-00030) Delphi consensus (Brescia Team, in progress) Master tracers’ practice and reliability (Brescia Team, in progr) Development of certification platform (Duchesne and coll) Validation data (Brescia Team – companion paper 1) Protocol definition (Brescia Team – companion paper 2) Validation vs pathology (TBD) DONEIN PROGRESSPLANNED

20. GOLDSTANDARD VALIDATION VS CURRENT PROTOCOLS ASSESSMENT OF SOURCES OF VARIANCE TRAINING SET DEVELOPMENT VALIDATION VS PATHOLOGY 20 naïve tracers 5 master tracers 1 tracer Local Protocol ADNI scans: 2 x 5 Scheltens’s atrophy scores x 2 sides x 2 magnet strengths (1.5-3T) Training ADNI scans: 10 at 1.5T x 2 sides x 7 SUs x 2 tracing rounds Total per rater: 40 hippos QUALIFICATION QUALIFICATION Harmonized Protocol ADNI scans: 2 x 5 Scheltens’s atrophy score x 2 sides x 2 magnet strengths (1.5-3T) Total per rater: 40 hippos Harmonized Protocol ADNI scans: 2 x 5 Scheltens’s atrophy score x 2 sides x 2 magnet strength (1.5-3T) Total per rater: 40 hippos Harmonized Protocol: Pathological datasets: Mayo Clinic and NYU Total: about 40 hippos Assessment of variance due to rater and center REFERENCE PROBABILISTIC MASKS with 95% C.I. Best 5 naïve tracers Assessment of agreement with volume on pathology or ex vivo MRI and correlation with neuronal density Harmonized Protocol ADNI scans: 2 sides x 5 Scheltens’s atrophy scores x 3 time points (bl-1y-2y) x 3 scanners (+ retracing @ bl) x 2 magnet strengths (1.5-3T) Total per rater: 240 hippos Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater TRAINING SET

21. GANTT