Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation

1. Medial Visual Fragment Image Representation for Perceptual Organization and Segmentation Amir Tamrakar and Benjamin Kimia LEMS, Brown University POCV 2004, Washington DC

2. Main Theme • Developed a novel intermediate representation of images, the Medial Visual Fragment representation that • Encodes simultaneously both contour and region properties. • Can deal with open contours and semi-closed regions • Represents fragments of objects or surfaces for which there is evidence. • Developed a language in terms of the shock graph suitable for perceptual reasoning with these visual fragments. • This talk is about a suitable intermediaterepresentation and not about an algorithm for segmentation. • Basically, this representation • Encodes spatial relationships between contours and regions • Improves the notion of good continuation of object contours by including region properties with it

3. Goal of Segmentation • To partition the image into units corresponding to perceived objects or surfaces of objects • Traditionally, this has involved segmenting it into a non-overlapping set of fragments. This “jigsaw puzzle” like segmentation doesn’t suffice to explain our perception. • Hence the term “Perceptual Organization” is often used to describe grouping of these fragments into units that agree with our perception. • A layered and hierarchical representation seems to be more appropriate = +

4. Gestalt Laws • For Perceptual Grouping, the Gestalt Laws of grouping are still the de facto theory to use. • Laws of similarity and good continuity are the most popular. • Continuity of Contours • Continuity of Surfaces • Continuity of volumes From Peter Tse, 1999

5. Boundaries vs. Regions • Thus, two kinds of cues are available from an image for the presence of objects • Boundaries • Locations of discontinuities of surface properties. • Interior Regions • Cohesive regions due to similarities in surface properties. • These cues are related and often consider duals. • A closed boundary encloses a region and the perimeter of a region is a boundary. • However, not all boundaries are closed and not all regions are bounded by contours. • In the past, people have often worked with them separately but people have realized the need to combine them • Humans segmentation makes use of both these information (Fowlkes, Martin and Malik, ’03)

6. Use of Boundaries in Segmentation • Local differential operators are used to locate them (edgels) • Hence, the information is inherently noisy. • These edgels are grouped into long smooth curve segments on the basis of good continuation. • Shashua and Ullman, Alter and Basri, Guy and Medioni, Harold and Horaud, Sarkar and Boyer, Williams and Thornber etc. • A closed curve is more salient that an open one • Kovacz and Julesz, Elder, Williams, etc. From Shashua and Ullman, 1988

7. Use of Boundaries in Segmentation • Often times boundary fragments are forced to join others for producing a segmentation with rich topology instead of the low-order connectivity that most people look for (Rothwell, Mundy Hoffman,1995). Using the VanDuc edge detector and linker, (Rothwell, Mundy, Hoffman, Nguyen, 95)

8. Contour Continuity is NOT Enough. It’s object fragment continuity that one really cares about.

9. Surface Continuity • Surface continuity on at least one side of the contour seems desirable. Surface Does not Continue Surface Continues

10. Surface Continuity Resolves Ambiguities

11. Fundamental Problem for Contours • There is no sense of spatial relationship between contours. • One cannot determine how far out from the curve one should venture to collect the required surface information. • In other words, contours do not code the extent of the object/surface that they bound unless they are closed. • Contour based grouping would benefit from having this kind of spatial information.

12. Another Problem • Some very salient contours are not closed because they arise due to 3D structure (folds joining the body). • They necessarily terminate (cusps) • Traditionally, these open contours are discarded and a lot of structural information is lost with it. • A segmentation scheme would tremendously benefit by having a representation that allows for such open contours. From Zucker et al

13. Use of Regions in Segmentation • Pixels are grouped into atomic patches on the basis of the similarity of their surface properties like intensity, color, texture, etc. • Adjacent patches are grouped to form larger regions e.g., Seeded Region Growing, etc. • As a graph partitioning problem e.g. Normalized Cuts (Shi and Malik), etc. • Multigrid-based Segmentation by Weighted Aggregation (SWA) (Sharon et al) SWA segmentation, From Sharon et al, 2001

14. Common Problems • The boundaries of these regions may or may not be meaningful as contours of objects. • A large portion arises merely due to competition between adjacent regions as they try to grow. • They are especially problematic if there are gradual variations in intensity (shading) or texture • Integrating over the patch integrates this variation and exaggerates the difference between the patches • Leakage is a common problem. • The addition of contours and reasoning about their continuity allows for a splitting of the regions that have merged due to leakage.

15. Conclusion • One needs to reason with both kinds of information at the same time. • One thus requires an intermediate representation that can provide both kinds of information simultaneously. • One should also be able to represent open contours and regional properties associated with them.

16. The Proposed Intermediate Representation:Medial Visual Fragments

17. Motivation • We want to append on to the contour the regional information around it.

18. Motivation • We want to append on to the contour the regional information around it. But the question is how far off should one go from the contour?

19. Motivation • We want to append on to the contour the regional information around it. Presumably, there are other contours that are trying to capture some region as well.

20. Motivation • We want to append on to the contour the regional information around it. The Best Answer is as far as the Medial Axis. Axis In absence of any other information, the medial axis is the bisector of two regions.

21. C+ Medial Axis Medial Fragment C- Medial Visual Fragment Representation • The Medial Axis “binds” together a pair of contours and the region between them.

22. Medial Visual Fragment Representation Definition: • In the grassfire analogy of Blum, the burnt region corresponding to the each medial axis segment is a Medial Visual Fragment. • i.e., it is the union of all pairs of rays (PP+, PP-) arising from all points along the segment. C+ Medial Axis Medial Fragment C-

23. Medial Visual Fragment Representation Proposition 1: • An image with an associated contour map (a set of curve segments) is partitioned into a set of medial visual fragments.

24. Medial Visual Fragment Representation Proposition 1: • An image with an associated contour map (a set of curve segments) is partitioned into a set of medial visual fragments. • Every point P in the image, there exists a shock segment k described by a curve γk parameterized by arclength s with a local coordinate system of axis tangent/normal (T(s), N(s)) and velocity v(s) such that for some t Є [0, r(s)], • Proposition 2: • These fragments satisfy the segmentation criterion.

25. Medial Visual Fragment Representation Average intensity computed at each Medial Fragment Original Image with its contour fragments The Medial Axis computed from these contour fragments The Medial Fragments

26. Medial Visual Fragment Representation • Medial Visual Fragments formed by various configurations of curve-pairs. • Between two open contours • Between one single open contour • Enclosed by a single closed contour • Between a pair of closed contours D A B C

27. Strengths of Our Representation • Allows for a combined region and contour description explicitly. C+ Medial Axis C- Visual Fragment

28. Strengths of Our Representation • Allows for open contours and semi-closed regions.

29. Strengths of Our Representation • Adaptively partitions the region around an open or closed contour into “influence zones” for gathering regional information around the contour.

30. Strengths of Our Representation • Allows for reasoning about fragment continuity in terms of “skeletal continuity” (Continuity of a pair of contours and the region between them).

31. Perceptual Organization using Visual Fragments

32. Perceptual Organization using Visual Fragments • Philosophy: • This representation ties PO and Recognition • We have proposed before that Perceptual Organization is only one half of Recognition (POCV 01) • The process of PO is a process of Perceptual Reasoning. • We have developed a Language in which to perform this type of reasoning • The language is that of transformations of the shock graph.

33. Image Medial Visual Fragments Medial Axis Transforms Perceptual Organization • The transformations on the representation transform the underlying image domain as well. From POCV ‘01

34. Perceptual Organization using Visual Fragments • Perceptual Organization is accomplished by Transforming the Medial Axis and hence the Medial Visual Fragments. • The viability of the transformed image defines cost of the transformation. • The Canonical Transforms on the Medial Axis are: • Gap Transform • Loop Transform • All the operations required for segmentation/perceptual grouping can be described as compositions of the canonical Medial Axis Transforms. • The choice of the optimal sequence of transformations is the process of perceptual Organization.

35. Medial Visual Fragment Transforms: Gap Transform • Gaps in the contours produce “degenerate” Medial Axis segments (i.e. arising from a pair of points) • The removal of such a segment (Gap Transform) closes the gap by linking the contour fragments. Completion Curve Post Gap Transform Contours with a gap Gap Axis segment

36. Medial Visual Fragment Transforms: Gap Transform • Ingredients for a viable gap transform: • Fragments A & B go together AND/OR fragments C & D go together • Reasonable curve completion is possible between C1 and C2.

37. Medial Visual Fragment Transforms: Gap Transform • Post gap transform: • The curves C1 and C2 have been connected by the completion curve • The Medial Visual Fragments have merged

38. Medial Visual Fragment Transforms: Gap Transform • Different varieties of Gap Transforms: • Completion assisted by another contour • Completion assisted by contours on either side • To Form Junctions

39. Medial Visual Fragment Transforms: Gap Transform From Berkeley Segmentation Database

40. Medial Visual Fragment Transforms: Gap Transform From Berkeley Segmentation Database

41. Medial Visual Fragment Transforms Loop Transform Motivation:

42. Medial Visual Fragment Transforms Loop Transform Motivation: An intervening curve fragment will claim its territory preventing C1 and C2 from talking to one another.

43. Medial Visual Fragment Transforms Loop Transform Motivation: Internal Structure (e.g. fold) Surface Markings

44. Medial Visual Fragment Transforms Loop Transform Implementation: The reverse process is the Loop Transform.

45. Medial Visual Fragment Transforms Loop Transform New layer Implementation: + Denotes attachment • Lift element into a new layer attached to the current fragment • Fill in under it to reflect this removal.

46. + Medial Visual Fragment Transforms Loop Transform New layer Motivation: = Texture mappedon to it Main fragment

47. Medial Visual Fragment Transforms Loop Transform From Berkeley Segmentation Database

48. Medial Visual Fragment Transforms Loop Transform From Berkeley Segmentation Database

49. + Reasoning with Occlusion • An occlusion presents itself as a loop in the shock graph. • The removal of this loop “lifts” the occluder onto a new layer. • The occluded fragments can then be completed.

50. Perceptual Grouping Example • A torus being grouped behind the occluder