460 likes | 1.01k Views
02/25/10. Graph-based Segmentation. Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem. Last class. Gestalt cues and principles of organization Mean-shift segmentation Good general-purpose segmentation method Generally useful clustering, tracking technique
E N D
02/25/10 Graph-based Segmentation Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem
Last class • Gestalt cues and principles of organization • Mean-shift segmentation • Good general-purpose segmentation method • Generally useful clustering, tracking technique • Watershed segmentation • Good for hierarchical segmentation • Use in combination with boundary prediction
Today’s class • Treating the image as a graph • Normalized cuts segmentation • MRFs Graph cuts segmentation • Recap • Go over HW2 instructions
Images as graphs • Fully-connected graph • node for every pixel • link between every pair of pixels, p,q • similarity wij for each link i wij c j Source: Seitz
Similarity matrix Increasing sigma
Segmentation by Graph Cuts w • Break Graph into Segments • Delete links that cross between segments • Easiest to break links that have low cost (low similarity) • similar pixels should be in the same segments • dissimilar pixels should be in different segments A B C Source: Seitz
Cuts in a graph • Link Cut • set of links whose removal makes a graph disconnected • cost of a cut: B A • One idea: Find minimum cut • gives you a segmentation • fast algorithms exist for doing this Source: Seitz
Cuts in a graph B A • Normalized Cut • a cut penalizes large segments • fix by normalizing for size of segments • volume(A) = sum of costs of all edges that touch A Source: Seitz
Recursive normalized cuts • Given an image or image sequence, set up a weighted graph: G=(V, E) • Vertex for each pixel • Edge weight for nearby pairs of pixels • Solve for eigenvectors with the smallest eigenvalues: (D − W)y = λDy • Use the eigenvector with the second smallest eigenvalue to bipartition the graph • Note: this is an approximation 4. Recursively repartition the segmented parts if necessary http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf Details:
Normalized cuts: Pro and con • Pros • Generic framework, can be used with many different features and affinity formulations • Provides regular segments • Cons • Need to chose number of segments • High storage requirement and time complexity • Bias towards partitioning into equal segments • Usage • Use for oversegmentation when you want regular segments
Markov Random Fields Node yi: pixel label Edge: constrained pairs Cost to assign a label to each pixel Cost to assign a pair of labels to connected pixels
Markov Random Fields Unary potential • Example: “label smoothing” grid 0: -logP(yi = 0 ; data) 1: -logP(yi = 1 ; data) Pairwise Potential 0 1 0 0 K 1 K 0
Solving MRFs with graph cuts Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1)
Solving MRFs with graph cuts Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1)
Grab cuts and graph cuts Magic Wand(198?) Intelligent ScissorsMortensen and Barrett (1995) GrabCut User Input Result Regions Regions & Boundary Boundary Source: Rother
Colour Model Gaussian Mixture Model (typically 5-8 components) R R Iterated graph cut Foreground &Background Foreground G Background G Background Source: Rother
Foreground (source) Min Cut Background(sink) Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time Graph cutsBoykov and Jolly (2001) Image Source: Rother
Graph cuts segmentation • Define graph • usually 4-connected or 8-connected • Define unary potentials • Color histogram or mixture of Gaussians for background and foreground • Define pairwise potentials • Apply graph cuts • Return to 2, using current labels to compute foreground, background models
GrabCut – Interactive Foreground Extraction10 Moderately straightforward examples … GrabCut completes automatically
GrabCut – Interactive Foreground Extraction11 Difficult Examples Camouflage & Low Contrast Fine structure Harder Case Initial Rectangle InitialResult
Using graph cuts for recognition TextonBoost (Shotton et al. 2009 IJCV)
Using graph cuts for recognition Unary Potentials Alpha Expansion Graph Cuts TextonBoost (Shotton et al. 2009 IJCV)
Limits of graph cuts • Associative: edge potentials penalize different labels • If not associative, can sometimes clip potentials • Approximate for multilabel • Alpha-expansion or alpha-beta swaps Must satisfy
Graph cuts: Pros and Cons • Pros • Very fast inference • Can incorporate recognition or high-level priors • Applies to a wide range of problems (stereo, image labeling, recognition) • Cons • Not always applicable (associative only) • Need unary terms (not used for generic segmentation) • Use whenever applicable
Further reading and resources • Normalized cuts and image segmentation (Shi and Malik) http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf • N-cut implementation http://www.seas.upenn.edu/~timothee/software/ncut/ncut.html • Graph cuts • http://www.cs.cornell.edu/~rdz/graphcuts.html • Classic paper: What Energy Functions can be Minimized via Graph Cuts? (Kolmogorov and Zabih, ECCV '02/PAMI '04)
Line detection and Hough transform • Canny edge detector = smooth derivative thin threshold link • Generalized Hough transform = points vote for shape parameters • Straight line detector = canny + gradient orientations orientation binning linking check for straightness
Robust fitting and registration Key algorithm • RANSAC
Clustering Key algorithm • Kmeans
EM and Mixture of Gaussians Tutorials: http://www.cs.duke.edu/courses/spring04/cps196.1/.../EM/tomasiEM.pdfhttp://www-clmc.usc.edu/~adsouza/notes/mix_gauss.pdf
Segmentation • Mean-shift segmentation • Flexible clustering method, good segmentation • Watershed segmentation • Hierarchical segmentation from soft boundaries • Normalized cuts • Produces regular regions • Slow but good for oversegmentation • MRFs with Graph Cut • Incorporates foreground/background/object model and prefers to cut at image boundaries • Good for interactive segmentation or recognition
Next section: Recognition • How to recognize • Specific object instances • Faces • Scenes • Object categories • Materials