Prior-Based Segmentation Using Unlevel Sets

1. Unlevel-Sets: Geometry and Prior-based Segmentation Tammy Riklin-Raviv Nahum Kiryati Nir Sochen Tel Aviv University

2. Tammy Riklin-Raviv

3. Prior knowledge is the key to segmentation Input image without prior knowledge expected outcome

4. fidelity uniformity parsimony “Generic” prior knowledge • Gestalt principles of perception • Minimum description length • Mumford-Shah functional min [(fidelity to image) + λ (uniformity within segments) + υ (total edge length)] Ω: image domainC: edge setu: segmented imagef: observed image

5. Bi-level limit (Chan & Vese, 2001) fidelity and uniformity parsimony Generic prior

6. Bi-level limit (Chan & Vese, 2001) Generic prior Level-set formulation Level-set function embedded contour Heaviside function Solution by gradient descent Osher & Sethian, 1988 Drawing borrowed from J. Sethian

7. Generic prior is often not enough, ... but a reference object (shape prior) can help! • Leventon, Grimson & Faugeras (CVPR’2000) • Tsai, Yezzi, Wells, Tempany, Tucker, Fan, Grimson & Willsky (CVPR’2001) • Chen, Thiruvenkadam, Tagare, Huang, Wilson & Geiser (VLSM’2001) • Rousson & Paragios (ECCV’2002) • Cremers, Kohlberger & Schnörr (ECCV’2002) Minimize

8. Shape prior The main issue: Shape Variability Typical approach • Collection of images of the reference object • Stochastic characterization of the reference object • Shape-term pushes the segmentation towards “likely” shape

9. But what about perspective?? Shape prior The main issue: Shape Variability Typical approach • Collection of images of the reference object • Stochastic characterization of the reference object • Shape-term pushes the segmentation towards “likely” shape

10. Our approach • Single prior image of the reference object • Deterministic representation of the shape prior • Explicitly account for perspective distortion Shape prior

11. The Variational & Level Set Approach meets Vision Geometry

12. Cone of Rays The cone of rays with vertex at the camera center. An image is obtained by intersecting this cone with a plane. A ray between a 3D point P and the camera centerccpierces the planes in the image pointspandp’. All Image points are related by planar Homography . Hartley & Zisserman 98

13. generalized cone un-level plane un-level set prior Representation of the Shape-Prior

14. Live illustration

15. In variational segmentation, evolve level-set function to make its zero-level set - a good segmentor, - similar to some “unlevel-set” of . Our concept

16. “unlevel set” of 0-level set of ~ • “Unlevel-set” of 0-level set of rotated and translated • Compare the Heaviside functions and Problem: Need to compare, within a variational framework, Possible Solution:

17. The 3D pose transformation 3X3 rotation matrix translation vector • tz ~ scale Minimize

18. Given and prior image image to segment and Construct initial level-set function generalized cone Initialize pose transformation parameters Algorithm: Initialization

19. u+ u- average“in” / “out” values as currently transformed level set function pose transformation parameters Algorithm: Iterate Compute Compute Update (gradient descent) Update (gradient descent)

20. Prior Input image Final level-set function Final contour Results

21. Results

22. Prior image Without shape prior Final level-set function Final contour on image Results Input image

23. Results Prior image Input image Without shape prior Generalized cone Final level-set function Final contour on image

24. Results Prior image Moved, rotated Input image Without shape prior With shape prior

25. Summary • Prior-based segmentation using level-sets • Single image of the reference object • Deterministic representation of the shape prior • Account for perspective distortion • Cope with occlusions