Simon K. Warfield, Ph.D. Associate Professor of Radiology Harvard Medical School

1. A Survey of Validation Techniques for Image Segmentation and Registration, with a focus on the STAPLE algorithm Simon K. Warfield, Ph.D. Associate Professor of Radiology Harvard Medical School

2. Outline • Validation of image segmentation • Overview of approaches • STAPLE • Validation of image registration • STAPLE algorithm available as open source software from: • http://www.nitrc.org/projects/staple • http://crl.med.harvard.edu/

3. Segmentation • Goal: identify or label structures present in the image. • Many methods: • Interactive or manual delineation, • Supervised approaches with user initialization, • Alignment with a template, • Statistical pattern recognition. • Applications: • Quantitative measurement of volume, shape or location of structures, • Provides boundary for visualization by surface rendering. Newborn MRI Segmentation.

4. Validation of Image Segmentation • Spectrum of accuracy versus realism in reference standard. • Digital phantoms. • Ground truth known accurately. • Not so realistic. • Acquisitions and careful segmentation. • Some uncertainty in ground truth. • More realistic. • Autopsy/histopathology. • Addresses pathology directly; resolution. • Clinical data ? • Hard to know ground truth. • Most realistic model.

5. Validation of Image Segmentation • Comparison to digital and physical phantoms: • Excellent for testing the anatomy, noise and artifact which is modeled. • Typically lacks range of normal or pathological variability encountered in practice. MRI of brain phantom from Styner et al. IEEE TMI 2000

6. Comparison To Higher Resolution MRI Photograph MRI Provided by Peter Ratiu and Florin Talos.

7. Comparison To Higher Resolution Photograph MRI Photograph Microscopy Provided by Peter Ratiu and Florin Talos.

8. Comparison to Autopsy Data • Neonate gyrification index • Ratio of length of cortical boundary to length of smooth contour enclosing brain surface

9. Staging Stage 3: at 28 w GA shallow indentations of inf. frontal and sup. Temp. gyrus (1 infant at 30.6 w GA, normal range: 28.6 ± 0.5 w GA) Stage 4: at 30 w GA 2 indentations divide front. lobe into 3 areas, sup. temp.gyrus clearly detectable (3 infants, 30.6 w GA ± 0.4 w, normal range: 29.9 ± 0.3 w GA) Stage 5: at 32 w GA frontal lobe clearly divided into three parts: sup., middle and inf. Frontal gyrus (4 infants, 32.1 w GA ± 0.7 w, normal range: 31.6 ± 0.6 w GA) Stage 6: at 34 w GA temporal lobe clearly divided into 3 parts: sup., middle and inf. temporal gyrus (8 infants, 33.5 w GA ± 0.5 w normal range: 33.8 ± 0.7 w GA) “Assessment of cortical gyrus and sulcus formation using MR images in normal fetuses”, Abe S. et al., Prenatal Diagn 2003 Stage 3 Stage 5 Stage 4 Stage 6

10. Neonate GI: MRI Vs Autopsy

11. GI Increase Is Proportional to Change in Age.

12. GI Versus Qualitative Staging

13. Neonate Gyrification

14. Validation of Image Segmentation • Comparison to expert performance; to other algorithms. • Why compare to experts ? • Experts are currently doing the segmentation tasks that we seek algorithms for. • Surgical planning. • Neuroscience research. • What is the appropriate measure for such comparisons ?

15. Measures of Expert Performance • Repeated measures of volume • Intra-class correlation coefficient • Spatial overlap • Jaccard: Area of intersection over union. • Dice: increased weight of intersection. • Vote counting: majority rule, etc. • Boundary measures • Hausdorff, 95% Hausdorff. • Bland-Altman methodology: • Requires a reference standard. • Measures of correct classification rate: • Sensitivity, specificity ( Pr(D=1|T=1), Pr(D=0|T=0) ) • Positive predictive value and negative predictive value (posterior probabilities Pr(T=1|D=1), Pr(T=0|D=0) )

16. Validation of Image Segmentation • STAPLE (Simultaneous Truth and Performance Level Estimation): • An algorithm for estimating performance and ground truth from a collection of independent segmentations.

17. STAPLE papers • Image segmentation with labels: • Warfield, Zou, Wells ISBI 2002 • Warfield, Zou, Wells MICCAI 2002. • Warfield, Zou, Wells, IEEE TMI 2004. • Commowick and Warfield IPMI 2009 • Image segmentation with boundaries: • Warfield, Zou, Wells MICCAI 2006. • Warfield, Zou, Wells PTRSA 2008. • Diffusion data and vector fields: • Commowick and Warfield IEEE TMI 2009

18. STAPLE: Estimation Problem • Complete data density: • Binary ground truth Ti for each voxel i. • Expert j makes segmentation decisions Dij. • Expert performance characterized by sensitivity p and specificity q. • We observe expert decisions D. If we knew ground truth T, we could construct maximum likelihood estimates for each expert’s sensitivity (true positive fraction) and specificity (true negative fraction):

19. Expectation-Maximization • Since we don’t know ground truth T, treat T as a random variable, and solve for the expert performance parameters that maximize: • Parameter values θj=[pj qj]T that maximize the conditional expectation of the log-likelihood function are found by iterating two steps: • E-step: Estimate probability of hidden ground truth T given a previous estimate of the expert quality parameters, and take expectation. • M-step: Estimate expert performance parameters by comparing D to the current estimate of T.

20. Probability Estimate of True Labels Estimate probability of tissue class in reference standard:

21. Binary Input: True Segmentation

22. Expert Performance Estimate p (sensitivity, true positive fraction) : ratio of expert identified class 1 to total class 1 in the image. q (specificity, true negative fraction) : ratio of expert identified class 0 to total class 0 in the image.

23. Newborn MRI Segmentation

24. Newborn MRI Segmentation Summary of segmentation quality (posterior probability Pr(T=t|D=t) ) for each tissue type for repeated manual segmentations. Indicates limits of accuracy of interactive segmentation.

25. Expert and Student Segmentations Test image Expert consensus Student 1 Student 2 Student 3

26. Phantom Segmentation Image Expert segmentation Student segmentations Image Expert Students Voting STAPLE

27. STAPLE Summary • Key advantages of STAPLE: • Estimates ``true’’ segmentation. • Assesses expert performance. • Principled mechanism which enables: • Comparison of different experts. • Comparison of algorithm and experts. • Extensions for the future: • Prior distribution or extended models for expert performance characteristics. • Estimate bounds on parameters.

28. Image registration • A metric: measures similarity of images given an estimate of the transformation. • Best metric depends on nature of the images. • Alignment quality ultimately possible depends on model of transformation. • The transformation is identified by solving an optimization problem. • Seek the transform parameters that maximize the metric of image similarity

29. Validation of Registration • Compare transformations • Take some images, apply a transformation to them. • Estimate the transform using registration • How well does the estimated transformation match the applied transform? • Check alignment of key image features • Fiducial alignment • Spatial overlap • Segment structures, assess overlap after alignment.

30. Intraoperative Nonrigid Registration • Fast: it should not take more than 1 min to make the registration. • Robust: the registration should work with poor quality image, artifacts, tumor... • Physics based: we are not only concerned in the intensity matching, but also interested in recovering the physical (mechanical) deformation of the brain. • Accurate: neuro-surgery needs a precise knowledge of the position of the structures. • Archip et al. NeuroImage 2007

31. Block Matching Algorithm Similarity measure: coefficient of correlation Divide a global optimization problem in many simple local ones Highly parallelizable, as blocks can be matched independently.

32. Block Matching Algorithm Displacement estimates are noisy.

33. Patient-specific Biomechanical Model Pre-operative image Automatic brain segmentation Brain finite element model (linear elastic)

34. Registration Validation • Landmark matching assessment in six cases • Parallel version runs in 35 seconds on a 10 dual 2GHz PC cluster • 7x7x7 block size • 11x11x25 window • 1x1x1 step • 50 000 blocks • 10 000 tetrahedra • 60 landmarks: • Average error = 0.75mm • Maximum error = 2.5mm • Data voxel size 0.8x0.8x2.5 mm3

35. Registration Validation • 11 prospective consecutive cases, • Alignment computed during the surgery. • Estimate of the registration accuracy – 95% Hausdorff distance of the edges of the registered preoperative MRI and the intraoperative MRI.

36. Automatic selection of fiducials (1)Non-rigid alignment of preoperative MPRAGE. Contours extracted from (1) with the Canny edge detector 95% Hausdorff metric computed Contours extracted from (2) with the Canny edge detector (2) Intraoperative whole brain SPGR at 0.5T

37. Alignment improvement

38. Visualization of aligned data • Matched preoperative fMRI and DT-MRI aligned with intraoperative MRI. Tensor alignment: Ruiz et al. 2000

39. Conclusion • Validation strategies for registration: • Comparison of transformations. • Fiducials • Manual, automatic. • Overlap statistics – as for segmentation. • Validation strategies for segmentation: • Digital and physical phantoms. • Comparison to domain experts. • STAPLE.

40. Neil Weisenfeld. Andrea Mewes. Richard Robertson. Joseph Madsen. Karol Miller. Michael Scott. Acknowledgements Collaborators • William Wells. • Kelly H. Zou. • Frank Duffy. • Arne Hans. • Olivier Commowick. • Alexandra Golby. • Vicente Grau. • This study was supported by: • R01 RR021885, R01 EB008015, R01 GM074068