Hierarchical Region-Based Segmentation by Ratio-Contour

1. Course Project of CSCE 790 Hierarchical Region-Based Segmentation by Ratio-Contour Jun Wang April 28, 2004

2. Background: Introduction of Image Segmentation a) What is image segmentation? Partition image (composed of pixels ) into several regions. Each region corresponds to meaningful structures or objects. b) Why do image segmentation? Representing Information and knowledge efficiently; Reduce the computation for further processing Important step for high level computer vision problem, such as object recognition and event recognition. C) Easy to do image segmentation?-----NO! Image is complicated: noise, texture, shadow, illumination, etc. Number of pixels is huge; Incorporation of prior knowledge;

3. Classification of Segmentation Methods • Low level methods Edge-based Region-based Clustering-analysis based Graph-theory approach • Middle level methods Active contour Level set methods • High level methods Model-based/Knowledge-guided Statistical-learning based method Template deformation

4. Graph Based Image Segmentation • Graph model of image 2D image ---- graph pixels, points from edges ---- vertexes in graph pixels pair or points pair ---- edges in graph pixel similarity, connection between points ---- weight of edge • segmentation problem transfer to graph problem image region segmentation ---- graph cut boundary detection in image ---- path search in graph

5. Graph Based Image Segmentation Methods • Graph cut based Minimum Cut (1993) Ratio Region Normalized Cut (1997) Ratio Cut (2002) • Cycle search based Shortest path algorithm (1996) Strongly connect component (2003) Ratio Contour (2003)

6. Background: Introduction of Ratio Contour Algorithm • Objective find Minimum Ratio Alternate (MRA) cycle in the graph, which is looked as the most salient boundary in the image. • Polynomial time computable solution for this cycle search problem.

7. Problem Formulation – obtain fragments image from original image Original image Canny detection Fragments image (after smoothing)

8. Problem Formulation – Construct a undirected graph for the smoothed fragments-image are vertices constructed from the endpoints of the fragments are edges consisting of two kinds of fragments: real fragments and virtual fragments

9. Definition of weight if is real fragment if is virtual fragment the curve from can be describe by function is the curvature at point is a constant related with the balance between smooth and proximity

10. Cost function of the boundary is a cycle in graph Define the cost function as: Optimize the cost function to obtain the most salient closed boundary with the minimum

11. Problem solution – Ratio Contour algorithm Reduction 1 Reduce the weight and length of solid edge to be zero by merging the weight and length of solid edge into the weight and length of adjacent dash edge. Reduction 2 Reduce the problem of finding a MRA cycle to the problem of finding a Negative total Weight Alternate (NWA) cycle. Reduction 3 Reduce the problem of finding a NWA cycle in the graph to the problem of finding a Minimum-Weight Perfect Matching (MWPM) in the same graph. Summary Finding WMPM cycle has classical polynomial time solution. And the reduction processes above are all polynomial time computable. ------ Details? Please refer to the reference.

12. Segmentation results on real images by Ratio Contour

13. Segmentation results on medical images by Ratio Contour

14. Further development solid-dash edges based graph construction to points based graph construction -----why? Fragments image Directed points image Directed points image (after smoothing) (with small length)

15. Further development curve length and curvature based weight definition to transition possibility based definition the edge weight by calculating the transition possibility from vertex to . by stochastic completion method. (Thornber and Williams 1996)

16. Further development undirected graph construction is developed in two step: • Construct directed graph based on previous weight calculation method. • Transfer directed graph to undirected graph by duplicating the vertex.

17. Boundary Based Segmentation 1. no region information is recorded; 2. the later detected boundary maybe traverse the form one; 3. may result in contradiction: same part belongs to different boundary inside area. 4. multi-boundary detection based segmentation can not be competent for hierarchical segmentation.

18. Example of Multi-boundary Detection

19. Region-based Segmentation 1.partition the graph after each iteration of boundary detection; 2. record the corresponding region information in Label Matrix; 3. the further boundary detection is executed on the generated multiple graphs, separately; 4. easy to extend for hierarchical segmentation.

20. Example of Region-based Segmentation

21. Example of Region-based Segmentation

22. Label Matrix – after three iteration of segmentation

23. Previous Example

24. When to stop? --- Threshold setting Set a threshold of the total weight of the detected boundary. If the weight is larger than the threshold, stop segmentation. • Segmentation methods two kind of methods for the hierarchical segmentation: Depth First Segmentation (DFS); Broad First Segmentation (BFS); The results are same.

25. Open boundary problem The boundaries traverse the other region or reach the edge of the image. It is not a completely closed boundary. • How to update the algorithm for necessary open boundary detection? The current boundary detection algorithm is for closed boundary, which must guarantee closure.

26. Boundary Vertex – special vertex in the graph Each extracted boundaries, include the original “boundary” - edge of image, are looked as a kind of complicated vertex. Each boundary is describe by a uniform vertex. • Property of Boundary Vertex 1. have length and shape; 2. every two points in the same boundary, the connection possibility is one. It means that the weight equals zero.

27. Illustration of Boundary Vertex Connection between two vertexand traverse the former extracted cycle boundary. How to calculate the weight between and ?

28. Calculating the weight as following. transition possibility from to , based on the definition of weight

29. Synthesized Experiments

31. Some Previous work (recently) S. Wang, T. Kubota, J. M. Siskind. Salient boundary detection using ratio contour, NIPS 2003 L.R. Williams & Thornber – using strongly connected component to extract boundary. J. H. Elder & S. W. Zuker – using shortest path algorithm to computing contour closure. • Related Work (by JoachimStahl) 1. contour intersection detection; 2. branch & band for solving counter intersection; • Ongoing Work 1. choose a efficient way for gap filling, especially large gap filling; 2. do more experiments on real images.

32. Some Reference 1) S. Wang, J. Wang, T. Kubota. From Fragments to Salient Closed Boundaries: An In-Depth Study, to appear in CVPR, Washington, DC, 2004 2) S. Wang, T. Kubota, J. M. Siskind, J. Wang. Salient Closed Boundary Extraction with Ratio Contour, under review 3) J. Elder and S. Zucker. Computing contour closure. In ECCV 1996 4) L. Williams and K.K. Thornber. A comparison measures for detecting natural shapes in cluttered background. IJCV 2000. 5) L. Williams and D. Jacobs. Stochastic completion fields: A neural model of illusory contour shape and salience. Neural Computation, 1997 6) S. Wang, T. Kubota, J. M. Siskind. Salient boundary detection using ratio contour, NIPS 2003 7) S. Mahamud, L.R. Williams, K. K. Thornber and K.Xu. Segmentation of multiple salient closed contours from real images. IEEE Transaction on PAMI, 2003. 8) Course notes of CSCE 790

33. Thank You!