Algorithms & Applications in Computer Vision

1. Algorithms & Applications in Computer Vision Lihi Zelnik-Manor Lecture 11: Structure from Motion

2. Image segmentation

3. The goals of segmentation • Group together similar-looking pixels for efficiency of further processing • “Bottom-up” process • Unsupervised “superpixels” X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.

4. The goals of segmentation • Separate image into coherent “objects” • “Bottom-up” or “top-down” process? • Supervised or unsupervised? human segmentation image Berkeley segmentation database:http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/

5. Water back Grass Tiger Tiger Sand head eye legs tail mouth shadow From Pixels to Perception outdoor wildlife

6. Berkeley Segmentation DataSet [BSDS] D. Martin, C. Fowlkes, D. Tal, J. Malik. "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics", ICCV, 2001

7. Contour detection ~2004 Local 7

8. Contour detection ~2008 (color) Global 8

9. Inspiration from psychology • The Gestalt school: Grouping is key to visual perception The Muller-Lyer illusion http://en.wikipedia.org/wiki/Gestalt_psychology

10. The Gestalt school • Elements in a collection can have properties that result from relationships • “The whole is greater than the sum of its parts” occlusion subjective contours familiar configuration http://en.wikipedia.org/wiki/Gestalt_psychology

11. Emergence http://en.wikipedia.org/wiki/Gestalt_psychology

12. Gestalt factors • These factors make intuitive sense, but are very difficult to translate into algorithms

13. Segmentation as clustering Source: K. Grauman

14. Two components to clustering • Similarity between pixels • For now lets assume it is the distance between RGB values. • The clustering algorithm

15. What is Clustering? • Organizing data into classes such that: • high intra-class similarity • low inter-class similarity • Finding the class labels and the number of classes directly from the data (in contrast to classification).

16. What is a natural grouping?

17. What is a natural grouping? Clustering is subjective Simpson's Family Females Males School Employees

18. What is Similarity?

19. Defining Distance Measures Definition: Let O1 and O2 be two objects from the universe of possible objects. The distance (dissimilarity) between O1 and O2 is a real number denoted by D(O1,O2) Peter Piotr 0.23 3 342.7

20. Segmentation as clustering Distance based on color only Source: K. Grauman

21. Partitional Clustering • Nonhierarchical, each instance is placed in exactly one of K nonoverlapping clusters. • Since only one set of clusters is output, the user normally has to input the desired number of clusters K.

22. Squared Error 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 Objective Function

23. Algorithmk-means • 1. Decide on a value for k. • 2. Initialize the k cluster centers (randomly, if necessary). • 3. Decide the class memberships of the N objects by assigning them to the nearest cluster center. • 4. Re-estimate the k cluster centers, by assuming the memberships found above are correct. • 5. If none of the N objects changed membership in the last iteration, exit. Otherwise goto 3.

24. k3 k1 k2 K-means Clustering: Step 1 Distance Metric: Euclidean Distance 5 4 3 2 1 0 0 1 2 3 4 5

25. k2 k1 k3 K-means Clustering: Step 2 Distance Metric: Euclidean Distance 5 4 3 2 1 0 0 1 2 3 4 5

26. k2 k1 k3 K-means Clustering: Step 3 Distance Metric: Euclidean Distance 5 4 3 2 1 0 0 1 2 3 4 5

27. k2 k1 k3 K-means Clustering: Step 4 Distance Metric: Euclidean Distance 5 4 3 2 1 0 0 1 2 3 4 5

28. k1 k2 k3 K-means Clustering: Step 5 Distance Metric: Euclidean Distance

29. Segmentation as clustering • K-means clustering based on intensity or color is essentially vector quantization of the image attributes • Clusters don’t have to be spatially coherent Image Intensity-based clusters Color-based clusters

30. Segmentation as clustering Distance based on color and position Source: K. Grauman

31. Segmentation as clustering • Clustering based on (r,g,b,x,y) values enforces more spatial coherence

32. K-Means for segmentation • Pros • Very simple method • Converges to a local minimum of the error function • Cons • Memory-intensive • Need to pick K • Sensitive to initialization • Sensitive to outliers • Only finds “spherical” clusters

33. Mean shift clustering and segmentation • An advanced and versatile technique for clustering-based segmentation http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.

34. Mean shift algorithm • The mean shift algorithm seeks modes or local maxima of density in the feature space Feature space (L*u*v* color values) image

35. Mean shift Searchwindow Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

36. Mean shift Searchwindow Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

37. Mean shift Searchwindow Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

38. Mean shift Searchwindow Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

39. Mean shift Searchwindow Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

40. Mean shift Searchwindow Center of mass Mean Shift vector Slide by Y. Ukrainitz & B. Sarel

41. Mean shift Searchwindow Center of mass Slide by Y. Ukrainitz & B. Sarel

42. Mean shift clustering • Cluster: all data points in the attraction basin of a mode • Attraction basin: the region for which all trajectories lead to the same mode Slide by Y. Ukrainitz & B. Sarel

43. Mean shift clustering/segmentation • Find features (color, gradients, texture, etc) • Initialize windows at individual feature points • Perform mean shift for each window until convergence • Merge windows that end up near the same “peak” or mode

44. Mean shift segmentation results http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

45. More results

46. More results

47. Mean shift pros and cons • Pros • Does not assume spherical clusters • Just a single parameter (window size) • Finds variable number of modes • Robust to outliers • Cons • Output depends on window size • Computationally expensive • Does not scale well with dimension of feature space

48. j wij i Images as graphs • Node for every pixel • Edge between every pair of pixels (or every pair of “sufficiently close” pixels) • Each edge is weighted by the affinity or similarity of the two nodes Source: S. Seitz

49. j wij i Segmentation by graph partitioning • Break Graph into Segments • Delete links that cross between segments • Easiest to break links that have low affinity • similar pixels should be in the same segments • dissimilar pixels should be in different segments A B C Source: S. Seitz

50. Measuring affinity • Suppose we represent each pixel by a feature vector x, and define a distance function appropriate for this feature representation • Then we can convert the distance between two feature vectors into an affinity with the help of a generalized Gaussian kernel: