Perceptual Organization: Segmentation and Optical Flow

1. Perceptual Organization: Segmentation and Optical Flow

2. Inspiration from psychology • The Gestalt school: Grouping is key to visual perception • “The whole is greater than the sum of its parts” occlusion subjective contours familiar configuration http://en.wikipedia.org/wiki/Gestalt_psychology

3. Gestalt grouping factors

4. Emergence http://en.wikipedia.org/wiki/Gestalt_psychology

5. Motion and perceptual organization • Even “impoverished” motion data can evoke a strong percept YouTube video G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.

6. Image segmentation

7. The goals of segmentation • Obtain primitives for other tasks • Perceptual organization, recognition • Graphics, image manipulation

8. Goal 1: Primitives for other tasks • Group together similar-looking pixels for efficiency of further processing • “Bottom-up” process • Unsupervised “superpixels” X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.

9. Segments as primitives for recognition • Image parsing or semantic segmentation: J. Tighe and S. Lazebnik, ECCV 2010, IJCV 2013

10. Goal 2: Recognition • Separate image into coherent “objects” • “Bottom-up” or “top-down” process? • Supervised or unsupervised? human segmentation image Berkeley segmentation database:http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/

11. Goal 3: Image manipulation • Interactive segmentation for graphics

12. Approaches to segmentation • Segmentation as clustering • Segmentation as graph partitioning • Segmentation as labeling

13. Segmentation as clustering Source: K. Grauman

14. Segmentation as clustering • K-means clustering based on intensity or color is essentially vector quantization of the image attributes • Clusters don’t have to be spatially coherent Image Intensity-based clusters Color-based clusters

15. Segmentation as clustering Source: K. Grauman

16. Segmentation as clustering • Clustering based on (r,g,b,x,y) values enforces more spatial coherence

17. j wij i Segmentation as graph partitioning • Node for every pixel • Edge between every pair of pixels (or every pair of “sufficiently close” pixels) • Each edge is weighted by the affinity or similarity of the two nodes Source: S. Seitz

18. Measuring affinity • Represent each pixel by a feature vector x and define an appropriate distance function Role of σ large σ small σ

19. j wij i Segmentation as graph partitioning • Break Graph into Segments • Delete links that cross between segments • Easiest to break links that have low affinity • similar pixels should be in the same segments • dissimilar pixels should be in different segments A B C Source: S. Seitz

20. Graph cut • Set of edges whose removal makes a graph disconnected • Cost of a cut: sum of weights of cut edges • A graph cut gives us a segmentation • What is a “good” graph cut and how do we find one? B A Source: S. Seitz

21. Minimum cut • We can do segmentation by finding the minimum cut in a graph • Efficient algorithms exist for doing this Minimum cut example

22. Minimum cut • We can do segmentation by finding the minimum cut in a graph • Efficient algorithms exist for doing this Minimum cut example

23. Cuts with lesser weight than the ideal cut Ideal Cut Normalized cut • Drawback: minimum cut tends to cut off very small, isolated components * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

24. Normalized cut • To encourage larger segments, normalize the cut by the total weight of edges incident to the segment • The normalized cut cost is: • Intuition: big segments will have a large w(A,V), thus decreasing ncut(A, B) • Finding the globally optimal cut is NP-complete, but a relaxed version can be solved using a generalized eigenvalue problem w(A, B) = sum of weights of all edges between A and B J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000

25. Normalized cut: Algorithm • Let W be the affinity matrix of the graph (n x n for n pixels) • Let D be the diagonal matrix with entries D(i, i) = Σj W(i, j) • Solve generalized eigenvalue problem (D − W)y = λDyfor the eigenvector with the second smallest eigenvalue • The ith entry of y can be viewed as a “soft” indicator of the component membership of the ithpixel • Use 0 or median value of the entries of y to split the graph into two components • To find more than two components: • Recursively bipartition the graph • Run k-means clustering on values of several eigenvectors

26. Example result

27. Challenge • How to define affinities for segmenting highly textured images?

28. Segmenting textured images • Convolve image with a bank of filters • Find textons by clustering vectors of filter bank outputs Image Texton map Filter bank J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

29. Segmenting textured images • Convolve image with a bank of filters • Find textons by clustering vectors of filter bank outputs • Represent pixels by textonhistograms computed over neighborhoods at some “local scale” • Define affinities as similarities between local texton histograms J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

30. Pitfall of texture features • Possible solution: check for “intervening contours” when computing affinities J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

31. Results: Berkeley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

32. Berkeley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

33. Normalized cuts: Pro and con • Pro • Generic framework, can be used with many different features and affinity formulations • Con • High storage requirement and time complexity: involves solving a generalized eigenvalue problem of size n x n, where n is the number of pixels

34. Efficient graph-based segmentation • Runs in time nearly linear in the number of edges • Easy to control coarseness of segmentations • Results can be unstable • P. Felzenszwalb and D. Huttenlocher, Efficient Graph-Based Image Segmentation, IJCV 2004

35. Segmentation as labeling • Suppose we want to segment an image into foreground and background • Binary labeling problem Credit: N. Snavely

36. Segmentation as labeling • Suppose we want to segment an image into foreground and background • Binary labeling problem User sketches out a few strokes on foreground and background… How do we label the rest of the pixels? Source: N. Snavely

37. Binary segmentation as energy minimization • Define a labeling L as an assignment of each pixel with a 0-1 label (background or foreground) • Find the labeling L that minimizes smoothness term data term How similar is each labeled pixel to the foreground or background? Encourage spatially coherent segments Source: N. Snavely

38. { computed by creating a color model from user-labeled pixels : “distance” from pixel to background : “distance” from pixel to foreground Source: N. Snavely

39. Source: N. Snavely

40. Neighboring pixels should generally have the same labels • Unless the pixels have very different intensities : similarity in intensity of p and q = 10.0 = 0.1 Source: N. Snavely

41. Binary segmentation as energy minimization • For this problem, we can efficiently find the global minimum using the max flow / min cut algorithm • Y. Boykovand M.-P. Jolly, Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images, ICCV 2001 Source: N. Snavely

42. Recall: Stereo as energy minimization • Energy functions of this form can be minimized using graph cuts I2 D I1 W1(i) W2(i+D(i)) D(i) data term smoothness term • Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

43. GrabCut C. Rother, V. Kolmogorov, and A. Blake, “GrabCut” — Interactive Foreground Extraction using Iterated Graph Cuts, SIGGRAPH 2004