Learning to Segment

1. Learning to Segment Authors: Eran Borenstein and Shimon Ullman Presented by: Dan Wang Some slices are from Ira Kemelmacher-Shlizerman and Sharon Alpert.

2. Eran Borenstein • Position • Computer vision director at Videosurf ( a company).   • Leading the development of the face-detection and recognition technology • Visiting scientist at the Division of Applied Mathematics Brown University • Academic Background • Post-doctoral Fellow, MSRI and EECS UC Berkeley. • Ph.D. & M.Sc., Department of Computer Science and Applied Mathematics , Weizmann Institute of Science. • B.Sc., Electrical Engineering Department, Technion - Israel Institute of Technology. • Publication: • 2 ECCV papers & 2 CVPR papers • Main interests • machine and human vision, in particular object classification and recognition, figure-ground segmentation, the interaction between top-down and bottom-up visual processes and other problems related to perceptual organization. • Website: http://www.dam.brown.edu/people/eranb/

3. Shimon Ullman • Current Position • Samy and Ruth Cohn Professor of Computer Science  • Head, Department of Computer Science and Applied Mathematics  • The Weizmann Institute of Science, Rehovot, Israel • Academic background • 1970-1973 Hebrew University, Jerusalem B.Sc. in Mathematics, Physics and Biology • 1973-1977 M.I.T. Cambridge MA Ph.D. Elec. Engr. and Computer Science • Previous Positions • 1981-1985 MIT Associate Professor  • 1981-1985 Weizmann Inst. Associate Professor  • 1986-1993 M.I.T. Professor  • 1986-present Weizmann Inst. Professor • Papers: • 50+ papers on CVPR, ICML, NIPS, etc. • Website: http://www.wisdom.weizmann.ac.il/~shimon/

4. Abstract • We describe a new approach for learning to perform class-based segmentation using only unsegmented training examples. • As in previous methods, we first use training images to extract fragments that contain common object parts. We then show how these parts can be segmented into their figure and ground regions in an automatic learning process. This is in contrast with previous approaches, which required complete manual segmentation of the objects in the training examples.

5. The figure-ground learning combines top-down and bottom-up processes and proceeds in two stages, an initial approximation followed by iterative refinement. The initial approximation produces figure-ground labeling of individual image fragments using the unsegmented training images. It is based on the fact that on average, points inside the object are covered by more fragments than points outside it. The initial labeling is then improved by an iterative refinement process, which converges in up to three steps. At each step, the figure-ground labeling of individual fragments produces a segmentation of complete objects in the training images, which in turn induce a refined figure-ground labeling of the individual fragments. • In this manner, we obtain a scheme that starts from unsegmented training images, learns the figure-ground labeling of image fragments, and then uses this labeling to segment novel images. Our experiments demonstrate that the learned segmentation achieves the same level of accuracy as methods using manual segmentation of training images, producing an automatic and robust top-down segmentation.

6. 中文摘要 • 我们提出了一种新的方法，该方法仅利用未分割的训练样本，学习如何进行特定类(class-based)的分割。 • 和之前的方法一样，我们首先利用训练图片提取包含物体部件的碎片(fragment)。然后介绍如何通过一个自动学习过程，将这些片段分割为前景和背景区域。相比之下，之前工作则要求训练集合的完全手动分割。 • 本文的前背景学习融合了自顶向下和自底向上的过程，分两个步骤进行：初始的粗分割和之后的迭代改进。初始的分割是利用未分割的训练图片，学习产生单个图像碎片的前背景标定。这一步骤基于一个事实：平均而言，覆盖于物体上的fragment数目比覆盖在物体外的多。而后，初始的分割通过最多3步迭代改进，即可收敛。每一步中，各个fragment的前背景标定产生训练图片的完整物体分割；这些随即用以改进单个fragment标定。 • 通过这种方式，我们得到了一个由未标定的训练集开始，学习图像碎片的前背景标定，然后利用这种标定分割新的图片的策略。我们的实验表明，和使用手动分割图片进行学习的分割相比，本文学习到的分割取得了，同等水平的精度，产生了一种自动、鲁棒的自顶向下的分割。

7. Backgrounds • Goal of figure-ground segmentation: • Identify an object in the image and separate it from the background. • Bottom-up approach: • First segment the image into regions, relying on the image-based criteria: • gray level • texture uniformity of image regions • Smoothness and continuity of bounding contours • … • Then identify the image regions that correspond to a single object.

8. Problems (1/2) Scale III Scale II Scale I Input

9. Problems (2/2) • Shortcomings of bottom-up approach: • An object may be segmented into multiple regions. • Some of the regions may be incorrectly merge the object with its background. • Complementary approach –top-down segmentation, usingprior knowledge of the object • Shape, color, texture…

10. Basic idea in this paper • Introducing shape of the specific class into the segmentation Top-down Learn how to segment this type of images Result: Binary segmentation Background / foreground Training images

11. Basic idea in this paper

12. Basic framework

13. Previous works • Top-down approaches • Deformable templates • Active shape models (ASM) • Active contours(snakes) • All of these top-down segmentation schemes requires extensive manual intervention.

14. Main contributions of this paper • Top down approach • Automatic fragments labeling

15. Organization • Introduction • Constructing a fragment set (Feature extraction) • Leaning the fragments figure-ground segmentation • Image segmentation by covering fragments • Improving the figure-ground labeling of fragments • Results

16. What is “fragment”? • Fragments represent local structure of common object parts (such as nose, leg, neck region etc. for the class of the horses) • Fragments are used as shape primitives. • Fragment extraction (F.E.) • Uses unsegmented class and non-class training images to extract and store image fragments. • These fragments represent local structure of common object parts (such as a nose, leg, neck region etc. for the class of horses). • Fragment labeling (F.L.) • This stage is obtained manually in the past. • This paper learns the segmentation scheme from unsegmented training images. • Fragment detection (F.D.)

17. Randomly collect a large set of candid. fragments Select subset of informative fragments Figure-Background Labeling Detection and segmentation class non-class Top-Down E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

18. Randomly collect a large set of candid. fragments Select subset of informative fragments class non-class Constructing a fragment set (F.E.) Ullman, S., Sali, E., Vidal-Naquet, M.: A fragment based approach to object representation and classification. In: Proc. of 4th international workshop on visual form, Capri, Italy (2001) 85–100

19. Selecting Informative Fragments • Informative = likely to be detected in class compared with non-class images • Fragments are added to maximize the gain in mutual information • Highly overlapping • Well-distributed over the figure • The process produces a set of fragments that are more likely to be detected in class compared with non-class images.

20. Organization • Introduction • Constructing a fragment set (Feature extraction) • Leaning the fragments figure-ground segmentation • Image segmentation by covering fragments • Improving the figure-ground labeling of fragments • Results

21. Randomly collect a large set of candid. fragments Select subset of informative fragments Figure-Background Labeling Detection and segmentation class non-class Top-Down E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

22. Randomly collect a large set of candid. fragments Select subset of informative fragments Figure-Background Labeling Detection and segmentation class non-class Learning the fragments figure-ground segmentation E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

23. Learning the fragments figure-ground segmentation • The learning relies on two main criteria: • The degree of cover • Border consistency

24. Informative fragments Bottom-Up segmentation Degree of cover Segmenting Informative Fragments (degree of cover) Segmentation ? E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

25. Formulation • Fragment: F • Different regions in a fragment: R1, R2, …, Rn • rj is the degree of cover. The higher rj , the higher its likelihood to be a figure region. • The figure part is defined by: , set

26. Segmenting Informative Fragments (Border consistency) Fragment hits location Fragment Fragment hits Edge map of each fragment hit Consistent edges

27. Segmenting Informative Fragments (Border consistency) • Fragment hit: Hj • The image patches where the fragment was detected. • Hj (x, y):the grey level value of pixel (x,y) • Apply an edge detector in each fragment hit • Determine the boundary that optimally separates figure from background regions. • The class-specific edges, will be consistently present among hits. • Other edges are arbitrary and change from one hit to the other.

28. Three types of edges: • Border edge, separating the figure part of the fragment from its background part. (we are looking for) • Interior edge, the edge within the figure part. • Noise edge, which is arbitrary and can appear anywhere in the fragment hit. (background textures or artifacts coming from the edge detector).

29. Segmenting Informative Fragments (combine the border consistency and the degree of cover) Segmentation ? Likelihood determined by the degree of cover + = Overlap with the average Edges D(x,y) of fragment in all the training images. E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

30. Formulation • F : Fragment • P : figure part of F • Pc : background part of F • : boundary between P andPc where

31. Overall process

32. Organization • Introduction • Constructing a fragment set (Feature extraction) • Leaning the fragments figure-ground segmentation • Image segmentation by covering fragments • Improving the figure-ground labeling of fragments • Results

33. Randomly collect a large set of candid. fragments Select subset of informative fragments Figure-Background Labeling Detection and segmentation class non-class Top-Down E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

34. Segmenting a test image • For the overlapping pixels, each fragment votes for the classification of pixels it covers • Count the number of votes for figure and background • Check consistency test between the received labeling and labeling of each fragment E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

35. Organization • Introduction • Constructing a fragment set (Feature extraction) • Leaning the fragments figure-ground segmentation • Image segmentation by covering fragments • Improving the figure-ground labeling of fragments • Results

36. Randomly collect a large set of candid. fragments Select subset of informative fragments Figure-Background Labeling Detection and segmentation class non-class Top-Down E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

37. Detection and segmentation Improving the figure-ground labeling of fragments refine refine …

38. Improving the figure-ground labeling of fragments • Once the labeled fragments produce consistent covers that segment complete objects in the training images, a region’s degree of cover can be estimated more accurately. • Use the average number of times its pixels cover figure parts in the segmented training images, rather than the average number of times its pixels overlap with other detected fragments. • The refined degree of cover is then used to update the fragment’s figure-ground labeling…

39. Experiments • 3 classes • Horse heads (quantitatively) • Human faces (qualitatively) • Cars (qualitatively)

40. Horse head class • Datasets • A training set of 139 class images (size 32x36). • Three independent experiments: • In each experiment, fragments are extracted from 15 images randomly from the training set. • Two different sizes of the fragment set: • 100 • The 40 most informative fragments within 100. • 100 horse head images are manually labeled out of the 139 images.

41. Two types of tests • Compare the automatic labeling with manual figure ground labeling of individual fragments. • Compare the segmentation of complete objects: • Automatically labeled fragments • Manually labeled fragments • Bottom-up segmentation

42. Results

43. Results This Alg. labeling Manual labeling Bottom-up seg. E. Borenstein, S. Ullman. Learning to Segment. ECCV (3) 2004: 315-328

44. This Alg. labeling Manual labeling Bottom-up seg. This Alg. labeling Manual labeling Bottom-up seg. • The approach is general and can be applied to different classes.

45. Zoom-in to the results Top-down Bottom-up The boundaries are exact We did not get the horse as one piece The shape of the horse is captured The boundaries are not exact Need to make a combination of Top-Down & Bottom-Up approaches

46. Main contributions of this paper • Top down approach • Automatic fragments labeling

47. Thanks for your attention!