1 / 46

Image Segmentation Jianbo Shi Robotics Institute Carnegie Mellon University

Image Segmentation Jianbo Shi Robotics Institute Carnegie Mellon University. Cuts, Random Walks, and Phase-Space Embedding. Joint work with: Malik,Malia,Yu. Taxonomy of Vision Problems. Reconstruction: estimate parameters of external 3D world. Visual Control:

shae
Download Presentation

Image Segmentation Jianbo Shi Robotics Institute Carnegie Mellon University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Image SegmentationJianbo ShiRobotics InstituteCarnegie Mellon University Cuts, Random Walks, and Phase-Space Embedding Joint work with: Malik,Malia,Yu

  2. Taxonomy of Vision Problems • Reconstruction: • estimateparameters of external 3D world. • Visual Control: • visually guided locomotion and manipulation. • Segmentation: • partition I(x,y,t) into subsets of separate objects. • Recognition: • classes: face vs. non-face, • activities: gesture, expression.

  3. We see Objects

  4. Outline • Problem formulation • Normalized Cut criterion & algorithm • The Markov random walks view of Normalized Cut • Combining pair-wise attraction & repulsion • Conclusions

  5. Edge-based image segmentation • Edge detection by gradient operators • Linking by dynamic programming, voting, relaxation, … Montanari 71, Parent&Zucker 89, Guy&Medioni 96, Shaashua&Ullman 88 Williams&Jacobs 95, Geiger&Kumaran 96, Heitger&von der Heydt 93 - Natural for encoding curvilinear grouping - Hard decisions often made prematurely

  6. Region-based image segmentation • Region growing, split-and-merge, etc... - Regions provide closure for free, however, approaches are ad-hoc. • Global criterion for image segmentation • Markov Random Fields e.g. Geman&Geman 84 • Variational approaches e.g. Mumford&Shah 89 • Expectation-Maximization e.g. Ayer&Sawhney 95, Weiss 97 - Global method, but computational complexity precludes exact MAP estimation - Problems due to local minima

  7. Bottom line: It is hard, nothing worked well, use edge detection, or just avoid it.

  8. Global good, local bad. • Global decision good, local bad • Formulate as hierarchical graph partitioning • Efficient computation • Draw on ideas from spectral graph theory to define an eigenvalue problem which can be solved for finding segmentation. • Develop suitable encoding of visual cues in terms of graph weights. Shi&Malik,97

  9. Image segmentation by pairwise similarities • Image = { pixels } • Segmentation = partition of image into segments • Similarity between pixels i and j Sij = Sji 0 Sij • Objective: “similar pixels should be in the same segment, dissimilar pixels should be in different segments”

  10. i A Segmentation as weighted graph partitioning Pixels iI = vertices of graph G Edges ij = pixel pairs with Sij> 0 Similarity matrix S = [ Sij ] is generalized adjacency matrix di = Sj Sij degree of i vol A =Si Adi volume of A I i Sij j

  11. Cuts in a graph • (edge) cut = set of edges whose removal makes a graph disconnected • weight of a cut cut( A, B ) =Si A,j BSij • the normalized cut NCut( A,B ) = cut( A,B )( + ) 1 . vol A 1 . vol B

  12. Normalized Cut and Normalized Association • Minimizing similarity between the groups, and maximizing similarity within the groups can be achieved simultaneously.

  13. The Normalized Cut (NCut) criterion • Criterion min NCut( A,A ) Small cut between subsets of ~ balanced grouping NP-Hard!

  14. Some definitions

  15. Normalized Cut As Generalized Eigenvalue problem • Rewriting Normalized Cut in matrix form:

  16. More math…

  17. y2i i A y2i i A Normalized Cut As Generalized Eigenvalue problem • after simplification, we get

  18. Interpretation as a Dynamical System

  19. Interpretation as a Dynamical System

  20. Brightness Image Segmentation

  21. brightness image segmentation

  22. Results on color segmentation

  23. Malik,Belongie,Shi,Leung,99

  24. Motion Segmentation with Normalized Cuts • Networks of spatial-temporal connections:

  25. Results

  26. Results Shi&Malik,98

  27. Results

  28. Results

  29. Stereoscopic data

  30. Conclusion I • Global is good, local is bad • Formulated Ncut grouping criterion • Efficient Ncut algorithm using generalized eigen-system • Local pair-wise allows easy encoding and combining of Gestalt grouping cues

  31. Goals of this work • Better understand why spectral segmentation works • random walks view for NCut algorithm • complete characterization of the “ideal” case • ideal case is more realistic/general than previously thought • Learning feature combination/object shape model • Max cross-entropy method for learning Malia&Shi,00

  32. The random walks view • Construct the matrix P = D-1S D = S = • P is stochastic matrix Sj Pij = 1 • P is transition matrix of Markov chain with state space I p = [ d1 d2 . . . dn ]Tis stationary distribution S11 S12 S1n S21 S22 S2n . . . Sn1 Sn2 Snn d1 d2 . . . dn 1 . vol I

  33. Reinterpreting the NCut criterion NCut( A, A ) = PAA + PAA PAB = Pr[ A --> B | A ] under P, p • NCut looks for sets that “trap” the random walk • Related to Cheeger constant, conductivity in Markov chains

  34. Reinterpreting the NCut algorithm (D-W)y = mDy m1=0 m2 . . . mn y1 y2 . . . Yn Px = lx l1=1 l2 . . . ln x1 x2 . . . xn mk = 1 - lk yk = xk The NCut algorithm segments based on the second largest eigenvector of P

  35. So far... • We showed that the NCut criterion & its approximation the NCut algorithm have simple interpretations in the Markov chain framework • criterion - finds “almost ergodic” sets • algorithm - uses x2 to segment Now: • Will use Markov chain view to show when the NCut algorithm is exact, i.e. when P hasK piecewise constant eigenvectors

  36. Piecewise constant eigenvectors: Examples • Block diagonal P (and S) Eigenvectors S Eigenvalues • Equal rows in each block Eigenvectors P Eigenvalues

  37. Piecewise constant eigenvectors: general case • Theorem[Meila&Shi] Let P = D-1S with D non-singular and let D be a partition of I. Then P has K piecewise constant eigenvectors w.r.t D iff P is block stochastic w.r.t D and P non-singular. Eigenvectors P Eigenvalues

  38. Block stochastic matrices D = ( A1, A2, . . . AK ) partition of I • Pis block stochastic w.r.t D Sj As Pij = Sj As Pi’jfori,i’ in same segment As’ • Intuitively: Markov chain can be aggregated so that random walk over D is Markov • P= transition matrix of Markov chain over D

  39. Learning Image fijq normalize Modell Sij Pij Pij* Segmentation 0, j A 00, j A 1 . |A| Learning image segmentation • Targets Pij* = for i in segment A • Model Sij= exp( Sqlqfqij )

  40. J lq The objective function J = - Si I Sj I Pij* log Pij J = KL( P* || P ) where p0= and Pi j* =p0Pij* the flow i j • Max entropy formulation maxPij H( j | i ) s.t. <fijq>p0Pij = <fijq>p0Pij* for all q Convex problem with convex constraints at most one optimum • The gradient = <fijq> p0Pij* -<fijq> p0Pij 1 . |I| `1234567890Qwertyuiop[] Asdfghjkl;’ zxcvbnm,./ -=\ 1 . |I|

  41. Experiments - the features • IC - intervening contour fijIC = max Edge( k ) k (i,j) Edge( k ) = output of edge filter for pixel k Discourages transitions across edges • CL - colinearity/cocircularity fijCL =+ Encourages transitions along flow lines • Random features i k j Edgeness 2-cos(2ai)-cos(2aj) 1 - cos(al) 2-cos(2ai + aj) 1 - cos(a0) ai j i aj orientation

  42. Training examples lCL lIC

  43. Test examples Original Image Edge Map Segmentation

  44. Conclusions • Showed that the NCut segmentation method has a probabilistic interpretation • Introduced • a principled method for combining features in image segmentation • supervised learning and synthetic data to find the optimal combination of features Graph Cuts Generalized Eigen-system Markov Random Walks

More Related