Today: Image Segmentation

1. Today: Image Segmentation • Image Segmentation Techniques • Snakes • Scissors • Graph Cuts • Mean Shift • Wednesday (2/28) • Texture analysis and synthesis • Multiple hypothesis tracking • Final Project Presentations • Q: one two-hour slot or two one-hour slots? • March 12 or 14?

2. From Images to Objects • What Defines an Object? • Subjective problem, but has been well-studied • Gestalt Laws seek to formalize this • proximity, similarity, continuation, closure, common fate • see notes by Steve Joordens, U. Toronto

3. Image Segmentation • Many Approaches Proposed • color cues • region cues • contour cues • Today: Three Approaches • Mean Shift (color-based) • D. Comaniciu, P. Meer, Robust Analysis of Feature Spaces: Color Image Segmentation, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR'97), 1997, 750-755. • Normalized Cuts (region-based) • J. Shi and J. Malik, Normalized Cuts and Image Segmentation, IEEE Conf. Computer Vision and Pattern Recognition(CVPR), 1997 • Snakes and Scissors (contour-based) • M. Kass, A. Witkin, and D. Terzopoulos, Snakes: Active contour models. International Journal of Computer Vision. v. 1, n. 4, pp. 321-331, 1987. • E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp. 191-198, 1995

4. Color-Based Segmentation • Segmentation by Histogram Processing • Given image with N colors, choose K • Each of the K colors defines a region • not necessarily contiguous • Performed by computing color histogram, looking for modes • This is what happens when you downsample image color range, for instance in Photoshop

5. Finding Modes in a Histogram • How Many Modes Are There? • Easy to see, hard to compute

6. Mean Shift [Comaniciu & Meer] • Iterative Mode Search • Initialize random seed, and fixed window • Calculate center of gravity of the window (the “mean”) • Translate the search window to the mean • Repeat Step 2 until convergence

8. Mean-Shift

9. More Examples: http://www.caip.rutgers.edu/~comanici/segm_images.html

10. Region-Based Segmentation • We Want Regions • why not build this in as a constraint?

11. Images as Graphs [Shi & Malik] • Graph G = (V, E, W) • node for every pixel • edge between every pair of pixels, p,q • weight wpq for each edge • wpq measures similarity • similarity: difference in color and position q wpq w p

12. w Segmentation by Graph Cuts • Break Graph into Segments • Delete edges that cross between segments • Easiest to break edges that have low weight: • similar pixels should be in the same segments • dissimilar pixels should be in different segments A B C

13. Normalized Cut • a cut penalizes large segments • fix by normalizing for size of segments Vol(A) Vol(B) Cuts in a graph • Edge Cut • set of edges whose removal makes a graph disconnected • cost of a cut: B A

14. The Normalized Cut (NCut) criterion • Given a Graph G = (V, E, W) • Find A ½ V that minimizes NP-Hard!

15. Normalize Cut in Matrix Form

16. Eigenvalue Problem • After lot’s of math, we get: • This is a Rayleigh Quotient • Solution given by “generalized” eigenvalue problem: • Solved by converting to standard eigenvalue problem: • Subtleties • optimal solution is second smallest eigenvector • gives real result—must convert into discrete values of y

17. Interpretation as a Dynamical System • Weights are Springs • eigenvectors correspond to vibration modes

18. Interpretation as a Dynamical System • Weights are Springs • eigenvectors correspond to vibration modes

19. Color Image Segmentation

20. The formulation is different however • In Boykov approach, each pixel was connected to a label node • Boykov approach gives better optimality guarantees, but requires discrete labels Graph Cuts Formulations • We’ve seen graph cuts before • Stereo (Y. Boykov, O. Veksler, and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts)

21. Segmentation Using Edges • Edges  Segments • Spurious edges • Missing edges How could user-interaction help?

22. Snakes [Kass et al.]

23. Snakes: Energy Minimization • Ingredients: • A contour • An energy function • Try to make the contour snap to edges • minimize Esnake • Issues • contour representation • energy function • minimization algorithm

24. Snakes • Contour representation • splines [Kass et al., IJCV 89, original formulation] • discrete set of points [Williams & Shah, CVGIP 55(1), 92] • implicit function [Malladi et al., 95 (see readings] • Energy functional • Variations on edge term (gradient magnitude, laplacian) • Minor variations in smoothness terms (first/second order) • Volume expansion (balloon) or contraction • Deformable templates [Yuille, IJCV 92] • Learned shape models [Isard & Blake, others] • Minimization algorithm • Numerical Euler integration [e.g., Kass] • Greedy algorithms [e.g., Williams] • Level set evolution [e.g., Malladi]

25. Issues: Scale Lowpass Images Bandpass Images Pyramid snakes for coarse-to-fine processing

26. Problem with Snakes • You can’t control the shape evolution* • * to be fair: Kass et al described how you can add forces to push and pull the snake as desired, but still kind of kludgy

27. Intelligent Scissors [Mortensen & Barrett]

28. Intelligent Scissors • Approach answers a basic question • Q: how to find a path from seed to mouse that follows object boundary as closely as possible? • A: define a path that stays as close as possible to edges

29. Intelligent Scissors • Basic Idea • Define edge score for each pixel • edge pixels have low cost • Find lowest cost path from seed to mouse mouse seed • Questions • How to define costs? • How to find the path?

30. Defining Costs • Link Costs • Costs are assigned to a pair of adjacent pixels (a link) link p q fz(q) = 0 if Laplacian is 0 at q fz(q) = 1 otherwise fd(q) measures difference in gradient direction

31. Path Search • Graph Search Algorithm • Computes minimum cost path from seed to allotherpixels

32. Results