Computer Vision Research in SPL ( Segmentation )

1. Computer Vision Research in SPL( Segmentation ) Jul. 3rd 2012 Prof. Sang Uk LEE Signal Processing Lab. Seoul Nat’l Univ.

2. Introduction: SPL • Signal Processing Lab. (SPL) • Founded in 1983 • Produced 43 Ph.D / 99 M.S degrees • Published 112 international journal / 217 conf. papers • Research Topics • Dense visual correspondence • Discrete MRF model • Image segmentation

3. Topics • Dense visual correspondence • Stereo matching under tough condition • Optical flow estimation with new motion prior • Deformable object registration for medical imaging • Publications • Y. S. Heo, K. M. Lee, S. U. Lee, Robust Stereo Matching using Adaptive Normalized Cross Correlation, PAMI 2011 • K. J. Lee, D. Kwon, I. D. Yun, S. U. Lee, Optical Flow Estimation with Adaptive Convolution Kernel Prior on Discrete Framework, CVPR 2010 • Y. S. Heo, K. M. Lee, S. U. Lee, Simultaneous Color Consistency and Depth Map Estimation for Radiometrically Varying Stereo Images, ICCV 2009 • Y. S. Heo, K. M. Lee, S. U. Lee, Mutual Information based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space, CVPR 2009 • D. Kwon, K. J. Lee, I. D. Yun, S. U. Lee. Nonrigid Image Registration Using Dynamic Higher-Order MRF Model, ECCV 2008

4. Topics • Dense visual correspondence • Stereo matching under tough condition • Optical flow estimation with new motion prior • Deformable object registration for medical imaging • Publications • Y. S. Heo, K. M. Lee, S. U. Lee, Robust Stereo Matching using Adaptive Normalized Cross Correlation, PAMI 2011 • K. J. Lee, D. Kwon, I. D. Yun, S. U. Lee, Optical Flow Estimation with Adaptive Convolution Kernel Prior on Discrete Framework, CVPR 2010 • Y. S. Heo, K. M. Lee, S. U. Lee, Simultaneous Color Consistency and Depth Map Estimation for Radiometrically Varying Stereo Images, ICCV 2009 • Y. S. Heo, K. M. Lee, S. U. Lee, Mutual Information based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space, CVPR 2009 • D. Kwon, K. J. Lee, I. D. Yun, S. U. Lee. Nonrigid Image Registration Using Dynamic Higher-Order MRF Model, ECCV 2008

5. Topics • Discrete MRF model • Optimization using windows annealing • High-order priors • High-order likelihood • Publications • H. Y. Jung, K. M. Lee, S. U. Lee, Stereo Reconstruction using High-Order Likelihoods, ICCV 2011 • D. Kwon, K. J. Lee, I. D. Yun, S. U. Lee, Solving MRFs with Higher-Order Smoothness Priors Using Hierarchical Gradient Nodes, ACCV 2010 • H. Y. Jung, K. M. Lee, S. U. Lee, Toward Global Minimum through Combined Local Minima, ECCV 2008 • H. Y. Jung, K. M. Lee, S. U. Lee, Window Annealing over Square Lattice Markov Random Field, ECCV 2008 • D. Kwon, K. J. Lee, I. D. Yun, S. U. Lee. Nonrigid Image Registration Using Dynamic Higher-Order MRF Model, ECCV 2008

6. Topics • Discrete MRF model • Optimization using windows annealing • High-order priors • High-order likelihood • Publications • H. Y. Jung, K. M. Lee, S. U. Lee, Stereo Reconstruction using High-Order Likelihoods, ICCV 2011 • D. Kwon, K. J. Lee, I. D. Yun, S. U. Lee, Solving MRFs with Higher-Order Smoothness Priors Using Hierarchical Gradient Nodes, ACCV 2010 • H. Y. Jung, K. M. Lee, S. U. Lee, Toward Global Minimum through Combined Local Minima, ECCV 2008 • H. Y. Jung, K. M. Lee, S. U. Lee, Window Annealing over Square Lattice Markov Random Field, ECCV 2008 • D. Kwon, K. J. Lee, I. D. Yun, S. U. Lee. Nonrigid Image Registration Using Dynamic Higher-Order MRF Model, ECCV 2008

7. Topics • Image segmentation • Higher order Interactive segmentation • Segmentation for medical images • Publications • Sang Hyun Park, Soochahn Lee, Il Dong Yun, and Sang Uk Lee, Automatic Bone Segmentation in Knee MR images using a Coarse-to-Fine Strategy, SPIE MI 2012. • Soochahn Lee, Sang Hyun Park, Hackjoon Shim, Il Dong Yun, and Sang Uk Lee, Optimization of local shape and appearance probabilities for segmentation of knee cartilage in 3-D MR images, CVIU 2011. • T. H. Kim, K. M. Lee, S. U. Lee, Learning Full Pairwise Affinities for Spectral Segmentation, CVPR 2010 • T. H. Kim, K. M. Lee, S. U. Lee, Nonparametric Higher-Order Learning for Interactive Segmentation, CVPR 2010 • T. H. Kim, K. M. Lee, S. U. Lee, Generative Image Segmentation using Random Walks with Restart,ECCV 2008

8. Topics • Image segmentation • Higher order Interactive segmentation • Segmentation for medical images • Publications • Sang Hyun Park, Soochahn Lee, Il Dong Yun, and Sang Uk Lee, Automatic Bone Segmentation in Knee MR images using a Coarse-to-Fine Strategy, SPIE MI 2012. • Soochahn Lee, Sang Hyun Park, Hackjoon Shim, Il Dong Yun, and Sang Uk Lee, Optimization of local shape and appearance probabilities for segmentation of knee cartilage in 3-D MR images, CVIU 2011. • T. H. Kim, K. M. Lee, S. U. Lee, Learning Full Pairwise Affinities for Spectral Segmentation, CVPR 2010 • T. H. Kim, K. M. Lee, S. U. Lee, Nonparametric Higher-Order Learning for Interactive Segmentation, CVPR 2010 • T. H. Kim, K. M. Lee, S. U. Lee, Generative Image Segmentation using Random Walks with Restart,ECCV 2008

9. Today’s Talk • This talk introduces image segmentation research. • Research topics • Dense visual correspondence • Stereo matching under tough condition • Optical flow estimation with new motion prior • Deformable object registration for medical imaging • Discrete MRF model • Optimization using windows annealing • High-order priors • High-order likelihood • Image segmentation • Higher order Interactive segmentation • Segmentation for medical images

10. Higher order Interactive segmentation • T. H. Kim, K. M. Lee, S. U. Lee, Nonparametric Higher-Order Learning for Interactive Segmentation, CVPR 2010

11. Introduction: Segmentation • Goal of Image Segmentation • Partition an image into coherent groups • Interactive Segmentation: Major trend in Image Segmentation • With user interaction, such as the scribbles labeling the pixels • Achieve a desirable result • Difficult for any user to provide the appropriate user interaction Inputs: Image & Scribbles with Labels Outputs: Multi-label segments

12. Interactive Segmentation • Limitation • Overall quality mainly depends on user-inputs. • The result should be insensitive to the following conditions: • Where the seeds are positioned • How many seeds are used Input GC1 RW2 Input GC1 RW2 • 1 Graph Cuts (GC) [Boykov, ICCV’01] • 2 Random Walker (RW)[Grady., PAMI’06]

13. Proposed Segmentation Model • Motivation • By using higher-order interactions, we can alleviate sensitivity with respect to seed quantity & placement. • However, previous higher-order models still have limitations. Input Robust Pn3 Proposed Alg. Input Robust Pn3 Proposed Alg. • 3Robust PnModel (Robust Pn) [Kohli, CVPR’08]

14. Proposed Segmentation Model • Main Idea • Nonparametric higher-order learning for segmentation • The soft label consistency constraint that “likelihood of a pixel” should be similar to “likelihood of its corresponding region” is imposed. • Likelihood estimation in a multilayerframework • To estimate the pixel & region likelihoods, we design mutually complementary “quadratic cost functions” and minimize them “simultaneously”. + Single Pixel Layer Multiple Region Layers Segmentation

15. 1) Nonparametric Higher-Order Cue • Decision Rule of Segmentation • Each pixel xi (or region yk) is assigned one label lwith maximum likelihoodp( xi | l ) (or p( yn| l ) ). Posterior Likelihood Region Likelihoods - For Pixel-based Segmentation, Generative Model Pixel Likelihoods Higher-Order Cue - For Region-based Segmentation, Generative Model * The prior p(l) is uniform.

16. 2) Cost Function for Likelihood Estimation • Joint Minimization of Cost Functions (1) • For pixel likelihoods (at pixel-based layer X ), Higher-Order Term Pairwise Term Unary Term Higher-Order Term: “The likelihood of a pixel should be similar to the mean likelihood of its corresponding regions .” Pixel-based layer Kregion-based layers

17. 2) Cost Function for Likelihood Estimation • Joint Minimization of Cost Functions (2) • For region likelihoods (at k-th region-based layer Yk ), Estimated Unary Term Pairwise Term Unary Term Estimated Unary Term: “The likelihood of a region should be similar to the weighted sum of likelihoods of its inner pixels .” Pixel-based layer k-thregion-based layer

18. Overview of Proposed Algorithm • Using Multiple Over-Segmentations ( K=3 ) • Mean Shiftmethod with different parameters Cost Functions for Likelihoods Graph with 1 pixel-based layer & K region-based layers Optimization Simultaneously, minimize K+1 quadratic functions , , … , . For region layer , For region layer . For pixel layer ,

19. Overview of Proposed Algorithm • Example of Multiple Over-Segmentations Region-layer #1 Region-layer #2 Region-layer #3 Input Segmentation with region-layer#1 Segmentation with region-layer #1,#2 Segmentation with region-layer #1,#2,#3 Segmentation with none region-layer

20. Experimental Results • Quantitative Evaluation • Grabcut dataset (50 images) • R : Resulting segmentation • R’ : Ground truth segmentation • 1 Graph Cuts (GC) [Boykov, ICCV’01] • 2 Geodesic Distance (GD)[Bai, ICCV’07] • 3 Random Walker (RW)[Grady., PAMI’06] • 4 Robust Pn Model (Robust Pn)[Kohli., CVPR’08]

21. Example Segmentations on Grabcut dataset Input GC RW Robust Pn Proposed Alg.

22. Accuracy : • Sensitivity Check with respect to Seed Quantity & Placement Image (a) Initial Tri-map 100% of (a) Randomly, 1% of (a) GC RW Robust Pn Proposed Alg.

23. Segmentations in Natural Images Images RW Robust Pn Images RW Robust Pn Proposed Alg. Proposed Alg.

24. Segmentation for medical images • S. C. Lee, S. H. Park, H. J. Shim, I. D. Yun, S. U. Lee, Optimization of local shape and appearance probabilities for segmentation of knee cartilage in 3-D MR images, CVIU 2011

25. Introduction : medical image segmentation • Necessity of medical image segmentation • Organ segmentation is crucial element in monitoring and understanding of the progress of disease. • Segmentation has been mostly done totally by hand, or semi-automatic segmentation methods. Butit needs experts effort & much time. Semiautomatic Segmentation Manual Segmentation User given scribbles Graph cut Result Takes 2~3 hours Takes 30~50 minutes

26. Introduction : medical image segmentation • Difficulty of automatic segmentation • Inhomogeneous intensity according to modality • Low tissue contrast • Deformable and irregularly shape of organs

27. Main idea • Motivation • Same organs have common global shape, but each subject has their own deformations locally. • We find the deformations in local view, and enforce the shape smoothness in global view. • Main Idea • Based on prior knowledge of training set, adaptive local prior and global shape prior are considered. • Both the local prior and the global prior are incorporated into a single hierarchical structure MRF.

28. Overview of algorithm

29. Overview of algorithm

30. Overview of algorithm 1 2 3 N : # of training set Test patch . . . . . . . . . . 1 2 3 N . . . . . Extract the similar patches in each regions from the training set ( Reference patches )

31. Overview of algorithm Local characteristics (difference between FG/BG histogram) Adaptive local prior Shape prior (template) Appearance prior (intensity histogram) 1 2 3 N : # of training set Make adaptive prior based on the shape and appearance information Test patch . . . . . . . . . . 1 2 3 N . . . . . Extract the similar patches in each regions from the training set ( Reference patches )

32. Overview of algorithm For each reference patch, MRF graph is constructed. Adaptive prior Graph cut Optimization Smoothness

33. Overview of algorithm

34. Overview of algorithm

35. Overview of algorithm

36. Overview of algorithm Boundary is likely to have high gradient Overlapping regions are likely to be similar and smoothly connected, TRW-s Optimization

37. Overview of algorithm

38. Experiment Result • Measurement • Dice similarity coefficient : • Quantitative Result S : Segmentation result • R : Ground truth GP : Global prior based method • LP : Local prior based method • HMRF : Proposed method

39. Experiment Result • Bone segmentation in knee MR image

40. Experiment Result • Cartilage segmentation in knee MR image

41. Experiment Result • Hippocampus segmentation in brain MR images

42. Experiment Result • 3D Error map Most errors occur on vague regions, but there are difficult parts even if radiologists segment by manually.

43. Experiment Result • Pathological deformities ( large shape variation )

44. Thank you! • Questions • Comments • Suggestions