Feature-preserving Artifact Removal from Dermoscopy Images

1. Feature-preserving Artifact Removal from Dermoscopy Images Howard Zhou1, Mei Chen2, Richard Gass2, James M. Rehg1, Laura Ferris3, Jonhan Ho3, Laura Drogowski3 • 1School of Interactive Computing, Georgia Tech • 2Intel Research Pittsburgh • 3Department of Dermatology, University of Pittsburgh

2. Skin cancer and melanoma • Skin cancer : most common of all cancers

3. Skin cancer and melanoma • Skin cancer : most common of all cancers Hemangioma Basal Cell Carcinoma Compound nevus Seborrheic keratosis [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

4. Skin cancer and melanoma • Skin cancer : most common of all cancers • Melanoma : leading cause of mortality Hemangioma Melanoma Basal Cell Carcinoma Compound nevus Seborrheic keratosis Melanoma [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

5. Skin cancer and melanoma • Skin cancer : most common of all cancers • Melanoma : leading cause of mortality • Early detection significantly reduces mortality Hemangioma Melanoma Basal Cell Carcinoma Compound nevus Seborrheic keratosis Melanoma [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

6. Clinical View Dermoscopy view [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

7. Dermoscopy view Clinical view Dermoscopy • Skin surface microscopy • Improve diagnostic accuracy by 30% for trained, experienced physicians • Requires 5 or more years of experience • Computer-aided diagnosis (CAD) to assist less experienced physicians

8. Artifacts in dermoscopy images • Hair, air-bubbles,… • Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006]

9. Artifacts in dermoscopy images • Hair, air-bubbles,… • Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006]

10. Artifacts in dermoscopy images • Hair, air-bubbles,… • Interfering with computer-aided diagnosis Hair  lesion boundary [ Image courtesy of Grana et al. 2006]

11. Artifacts in dermoscopy images • Hair, air-bubbles,… • Interfering with computer-aided diagnosis Hair  lesion boundary [ Image courtesy of Grana et al. 2006]

12. Artifacts in dermoscopy images • Hair, air-bubbles,… • Interfering with computer-aided diagnosis Hair  lesion boundary Hair  pigmented network [ Image courtesy of Grana et al. 2006]

13. Previous work • Hair detection and tracing • Fleming et al. 1998 • Thresholding and averaging • “DullRazor”, Tim K. Lee et al. 1997 • Schmid et al. 2003 • Thresholding and inpainting • Paul Wighton et al. 2008 (right here in the conference)

14. Schmid et al. • Detection: thresholding • Removal: morphological operations

15. Schmid et al. • Thresholding  false detection • Accidental removal of diagnostic features Thresholding Schmid et al. 2003

16. Schmid et al. • Morphological operation (neighbors’ average) blurring Morphological operation Schmid et al. 2003

17. Feature-preserving artifact removal (FAR) • Detection: Explicit curve modeling • Removal: Exemplar-based inpainting Schmid et al. 2003 Our method (FAR)

18. FAR • Curve modeling  more accurate hair detection Thresholding Curve modeling Schmid et al. 2003 Our method (FAR)

19. FAR • Exemplar-based inpainting  preserving features Morphological operation Exemplar-based inpainting Curve modeling Thresholding Schmid et al. 2003 Our method (FAR)

20. FAR • Exemplar-based inpainting  preserving features Morphological operation Exemplar-based inpainting Curve modeling Thresholding Schmid et al. 2003 Our method (FAR)

21. FAR • Exemplar-based inpainting  preserving features Schmid et al. 2003 Our method (FAR)

22. FAR • Exemplar-based inpainting  preserving features Schmid et al. 2003 Our method (FAR)

23. FAR • Exemplar-based inpainting  preserving features Schmid et al. 2003 Our method (FAR)

24. Luminance difference  dark thin structure Line points Detection Threholding Line points Dermoscopy image Line points linking Line segments Exemplarpatches Curve fitting & intersection analysis Hair removed Mask Parameterized curves Exemplar-based inpainting System overview

25. Input dermoscopy image

26. Enhancing dark-thin structure • Luminosity channel in CIE L*u*v* • Difference b/a morphological closing [ Schmid-Saugeona et al. 2003, “Towards a computer-aided diagnosis system for pigmented skin lesions” ]

27. Detecting line points Curve B(t) [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

28. n(t) Detecting line points Cross section Curve B(t) n(t) f(x) [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

29. n(t) Detecting line points Cross section Curve B(t) n(t) f(x) [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

30. n(t) Detecting line points Cross section Curve B(t) f’ = 0 |f’’| large n(t) f(x) [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

31. n(t) Detecting line points Cross section Curve B(t) f’ = 0 |f’’| large n(t) f(x) n(t) : direction ┴ curve B(t) eigenvector corresponding to the maximum absolute eigenvalue of the local Hessian [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

32. Detecting line points n(t) [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

33. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

34. Linking line points • Link the neighboring points to get line segments (sets of ordered line points)

35. Fitting polynomial curves • A set of ordered points Pis P

36. Fitting polynomial curves • A set of ordered points Pis • Parametric curve P

37. Fitting polynomial curves • A set of ordered points Pis • Parametric curve P B(t)

38. Fitting polynomial curves • A set of ordered points Pis • Parametric curve • Minimize sum of squared distance P B(t)

39. Fitting polynomial curves • A set of ordered points Pis • Parametric curve • Minimize sum of squared distance • Linear system (can be solved by Gaussian elimination) P B(t)

40. Intersection analysis Link Line segment Hair intersection Line segments Handling hair intersection Configurations: ……

41. Before curve fitting and linking Line segments

42. After curve fitting and linking Parameterized curves

43. After curve fitting and linking Parameterized curves

44. After curve fitting and linking Hair mask

45. After curve fitting and linking Hair mask

46. Exemplar-based inpainting • Fill in with patches from the image itself • Patch ordering structure propagation. [ Image courtesy of Criminisi et al. 2003 ] [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

47. Exemplar-based inpainting • Fill in with patches from the image itself • Patch ordering structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

48. Exemplar-based inpainting • Fill in with patches from the image itself • Patch ordering structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

49. Exemplar-based inpainting • Fill in with patches from the image itself • Patch ordering structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

50. Exemplar-based inpainting • Fill in with patches from the image itself • Patch ordering structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]