Sketch-based Image-independent Editing of 3D Tumor Segmentations using Variational Interpolation

1. Sketch-based Image-independent Editing of 3D Tumor Segmentations using Variational Interpolation Frank Heckel1, Stefan Braunewell1, Grzegorz Soza2, Christian Tietjen2, Horst K. Hahn1 1 Fraunhofer MEVIS, Germany, 2Siemens AG, Healthcare Sector, Imaging & Therapy Division, Computed Tomography, Germany

2. Motivation Solution Results Outlook Conclusion • Segmentation is one of the essential tasks in medical image analysis • Many sophisticated automatic segmentation algorithms exist … • … which might fail in some cases (low contrast, noise, biological variability) • What to do? • Manual segmentation?  Takes too long • Different algorithm?  Might fail as well • Locally correct the error! • Why do we need efficient segmentation editing tools?

3. Motivation Solution Results Outlook Conclusion • Requirements: • Intuitive interaction in 2D – estimate the user’s intention in 3D • Local modifications • Real-time feedback • Provide a general tool (for different objects and modalities) • Be independent of the preceding automatic algorithm • The user expects the tool to allow him or her to correct all errors • With only a few steps! • The segmentation problems are typically hard (noise, low contrast, …) • Do not use the image! • What makes segmentation editing a difficult problem?

4. Solution Motivation Results Outlook Conclusion • Use methods known from object reconstruction • Contour-based representation  Can be treated as a point cloud • Reconstruct a smooth surface using variational interpolation • Segmentation Formulated as an Object Reconstruction Problem

5. Solution Motivation Results Outlook Conclusion • Hole-handling: • Recursively check the level of embedding • Holes have an odd level  Invert the sign of the normals • Segmentation Formulated as an Object Reconstruction Problem without hole-handling with hole-handling

6. Solution Motivation Results Outlook Conclusion • Sketch-based Editing in 2D User input Correction result Edited region Part containing the center of gravity

7. Solution Motivation Results Outlook Conclusion • We have to deal with imperfection: • Sketch-based Editing in 2D add remove add + remove replcae

8. Solution Motivation Results Outlook Conclusion • A correction might generate new “holes”: • Remove all contours whose level of embedding has changed • Sketch-based Editing in 2D

9. Solution Motivation Results Outlook Conclusion • Compute a correction depth • Reconstruct the new surfacebetween start and end • 3D Extrapolation using Variational Interpolation

10. Solution Motivation Results Outlook Conclusion • Making the correction local: • Dilate the edited region • Duplicate it to all slices of the reconstruction • Use new segmentation in this region only • 3D Extrapolation using Variational Interpolation Sphere volume: →

11. Solution Motivation Results Outlook Conclusion • Manual Correction Workflow • Corrections can be performed in any view • User can arbitrarily switch between views • Previously performed corrections should be part of the new surface • Keep all user-inputs and use them for reconstruction user-input(1st step, axial view) user-input(2nd step, sagittal view)

12. Solution Motivation Results Outlook Conclusion

13. Results Solution Outlook Conclusion • Data: 89 tumors in CT (lung nodules, liver metastases, lymph nodes) • Participants: 2 technical experts with 6+ years experience in tumor segmentation and assessment • Qualitative rating of the correction tool • Median: 4 steps (53s), Avg. time per step: 0.4s1 92.1% 1 Intel Xeon X5550 (2.66GHz), 12GB RAM, Windows 7 64-Bit, 4 cores used

14. Outlook Results Conclusion • Main problem currently: Contradictory user inputs • Reconstruction is currently too slow for large objects (like the liver) • Comparison the other methods • Strongly depends on the specific segmentation task and the experience and requirements of the users • Proposal: Segmentation editing challenge

15. Conclusion Outlook • Segmentation editing is an indispensable step in the segmentation process • Efficient editing in 3D is challenging • Sketching provides an intuitive interface for segmentation editing in 2D • We have proposed a general, efficient editing tool • 2D corrections are extrapolated to 3D using object reconstruction • Can be used for any 3D modality and any compact object

16. Thank you! frank.heckel@mevis.fraunhofer.de