1 / 122

Understanding Energy Minimization using Graph Cuts in Image & Video Segmentation

Learn about the concept of Energy Minimization using Graph Cuts, its application in Image & Video Segmentation, and the key algorithms and techniques involved. Explore the seminal work of Boykov and Jolly [ICCV 2001] and comprehend the dynamic Graph Cuts process for efficient segmentation solutions

charleshamilton
Download Presentation

Understanding Energy Minimization using Graph Cuts in Image & Video Segmentation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ICCV Tutorial 2007 Philip Torr Papers, presentations and videos on web.....

  2. Overview Recycling Solutions Illustrate on 2D graph cuts, generalization to 3D easy

  3. MRF for Image & Video Segmentation Seminal Work of Boykov and Jolly [ICCV 2001] EnergyMRF = Unary likelihood Contrast Term Uniform Prior (Potts Model) Maximum-a-posteriori (MAP) solution x*= arg min E(x) x Data (D) Unary likelihood Pair-wise Terms MAP Solution

  4. solve SA differences between A and B PB* Simpler problem A and B similar SB Dynamic Graph Cuts PA cheaper operation PB computationally expensive operation

  5. Maximum flow MAP solution First segmentation problem Ga difference between Ga and Gb residual graph (Gr) second segmentation problem updated residual graph G` Gb Our Algorithm

  6. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) Source (1) a1 a2 Sink (0)

  7. Energy Minimization using Graph cuts What really happens? Building the graph If we cut a link to source or sink this indicates label Source (1) a1 a2 Sink (0)

  8. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) =2a1 Source (1) 2 a1 a2 Sink (0)

  9. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1 Source (1) 2 a1 a2 5 Sink (0)

  10. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 Source (1) 2 9 a1 a2 5 4 Sink (0)

  11. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 Source (1) 2 9 a1 a2 2 5 4 Sink (0)

  12. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (1) 2 9 1 a1 a2 2 5 4 Sink (0)

  13. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (1) 2 9 t-edges (unary terms) 1 a1 a2 2 5 4 n-edges (pair-wise term) Sink (0)

  14. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (1) 2 9 a1 = 1 a2 = 1 1 a1 a2 2 Cost of st-cut = 11 5 4 EMRF(1,1) = 11 Sink (0)

  15. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (1) 2 9 a1 = 1 a2 = 0 1 a1 a2 2 Cost of st-cut = 8 5 4 EMRF(1,0) = 8 Sink (0)

  16. Energy Minimization using Graph cuts What really happens? Building the graph EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 + CONSTANT TERM K Source (1) 2 9 a1 = 1 a2 = 0 1 a1 a2 2 Cost of st-cut = 8 5 4 EMRF(1,0) = 8 Sink (0)

  17. Energy Minimization using Graph cuts Posiform EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 + CONSTANT TERM K Posiform is a multilinear polynomials in binary variables with positive coefficients

  18. Computing the st-mincut from Max-flow algorithms Source (0) • The Max-flow Problem • Edge capacity and flow balance constraints 2 9 • Notation • Residual capacity • (edge capacity – current flow) • Augmenting path 1 a1 a2 2 5 4 • Simple Augmenting Path based Algorithms • Repeatedly find augmenting paths and push flow. • Saturated edges constitute the st-mincut. • [Ford-Fulkerson Theorem] Sink (1)

  19. Reparametrization Source (0) Key Observation 9 + α 2 Adding a constant to both the t-edges of a node does not change the edges constituting the st-mincut. 1 a1 a2 2 4 + α 5 Sink (1) E (a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 E*(a1,a2 ) = E(a1,a2) + α(a2+ā2) = E(a1,a2) + α [a2+ā2 =1]

  20. Reparametrization, second type Source (0) Other type of reparametrization 9 + α 2 All reparametrizations of the graph are sums of these two types. 1 - α a1 a2 2 + α 5 + α 4 Sink (1) Both maintain the solution and add a constant α to the energy.

  21. Reparametrization • Nice result (easy to prove) • All other reparametrizations can be viewed in terms of these two basic operations. • Proof in Hammer, and also in one of Vlad’s recent papers.

  22. Graph Re-parameterization s flow/residual capacity 0/7 0/1 0/5 xi xj 0/9 0/2 0/4 t G original graph

  23. Graph Re-parameterization Edges cut s flow/residual capacity 5/2 1/0 0/7 0/1 Compute Maxflow 3/2 0/5 xi xj xi xj 0/12 0/9 st-mincut 2/0 4/0 0/2 0/4 t t Gr G residual graph original graph

  24. Update t-edgeCapacities s 5/2 1/0 3/2 xi xj 0/12 2/0 4/0 t Gr residual graph

  25. Update t-edgeCapacities s capacity changes from 7 to 4 5/2 1/0 3/2 xi xj 0/12 2/0 4/0 t Gr residual graph

  26. excess flow (e) = flow – new capacity = 5 – 4 = 1 Update t-edgeCapacities s capacity changes from 7 to 4 5/-1 1/0 3/2 xi xj edge capacity constraint violated! (flow > capacity) 0/12 2/0 4/0 t G` updated residual graph

  27. excess flow (e) = flow – new capacity = 5 – 4 = 1 add e to both t-edges connected to node i Update t-edgeCapacities s capacity changes from 7 to 4 5/-1 1/0 3/2 xi xj edge capacity constraint violated! (flow > capacity) 0/12 2/0 4/0 t G` updated residual graph

  28. Update t-edgeCapacities excess flow (e) = flow – new capacity s = 5 – 4 = 1 capacity changes from 7 to 4 5/0 1/0 add e to both t-edges connected to node i 3/2 xi xj edge capacity constraint violated! (flow > capacity) 0/12 2/1 4/0 t G` updated residual graph

  29. Update n-edgeCapacities s • Capacity changes from 5 to 2 5/2 1/0 3/2 xi xj 0/12 2/0 4/0 t residual graph Gr

  30. Update n-edgeCapacities s • Capacity changes from 5 to 2 • - edge capacity constraint violated! 5/2 1/0 3/-1 xi xj 0/12 2/0 4/0 t Updated residual graph G`

  31. Update n-edgeCapacities s • Capacity changes from 5 to 2 • - edge capacity constraint violated! • Reduce flow to satisfy constraint 5/2 1/0 3/-1 xi xj 0/12 2/0 4/0 t Updated residual graph G`

  32. Update n-edgeCapacities s • Capacity changes from 5 to 2 • - edge capacity constraint violated! • Reduce flow to satisfy constraint • causes flow imbalance! 1/0 5/2 2/0 excess xi xj 0/11 deficiency 2/0 4/0 t Updated residual graph G`

  33. Update n-edgeCapacities s • Capacity changes from 5 to 2 • - edge capacity constraint violated! • Reduce flow to satisfy constraint • causes flow imbalance! • Push excess flow to/from the terminals • Create capacity by adding α = excess to both t-edges. 1/0 5/2 2/0 excess xi xj 0/11 deficiency 2/0 4/0 t Updated residual graph G`

  34. Update n-edgeCapacities s • Capacity changes from 5 to 2 • - edge capacity constraint violated! • Reduce flow to satisfy constraint • causes flow imbalance! • Push excess flow to the terminals • Create capacity by adding α = excess to both t-edges. 5/3 2/0 2/0 xi xj 0/11 3/0 4/1 t Updated residual graph G`

  35. Update n-edgeCapacities s • Capacity changes from 5 to 2 • - edge capacity constraint violated! • Reduce flow to satisfy constraint • causes flow imbalance! • Push excess flow to the terminals • Create capacity by adding α = excess to both t-edges. 5/3 2/0 2/0 xi xj 0/11 3/0 4/1 t Updated residual graph G`

  36. Complexity analysis of MRF Update Operations *requires k edge update operations where k is degree of the node

  37. Improving the Algorithm • Finding augmenting paths is time consuming. • Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004] • Reuses search trees after each augmentation. • Empirically shown to be substantially faster. • Our Idea • Reuse search trees from previous graph cut computation • Saves us search tree creation tree time [O(#edges)] • Search trees have to be modified to make them consistent with new graphs • Constrain the search of augmenting paths • New paths must contain at least one updated edge

  38. Reusing Search Trees c’ = measure of change in the energy • Running time • Dynamic algorithm (c’ + re-create search tree ) • Improved dynamic algorithm (c’) • Video Segmentation Example - Duplicate image frames (No time is needed)

  39. Experimental Analysis Interactive Image segmentation (update unary terms) EnergyMRF= additional segmentation cues user segmentation cues static: 175 msec dynamic : 80 msec dynamic (optimized): 15 msec static : 175 msec

  40. ExperimentalAnalysis Image segmentation in videos (unary & pairwise terms) EnergyMRF= Image resolution: 720x576 static: 220 msec dynamic (optimized): 50 msec Dynamic Graph Cuts Graph Cuts

  41. ExperimentalAnalysis Image segmentation in videos (unary & pairwise terms) EnergyMRF= Image resolution: 720x576 static: 177 msec dynamic (optimized): 60 msec Dynamic Graph Cuts Graph Cuts

  42. ExperimentalAnalysis Running time of the dynamic algorithm MRF consisting of 2x105 latent variables connected in a 4-neighborhood.

  43. Some Conclusions • An exact dynamic algorithm for computing st-mincut • - handles arbitrary changes in the graph • Rapidly solves dynamic sub-modular energy functions • Running time roughly proportional to the number of changes in the energy terms.

  44. ObjCut and PoseCut: • Combination of Top Down and Bottom Up Cues.. • Classical Vision Problem. • Combine theory of MRF’s with Object Recognition

  45. Objective Aim to get a clean segmentation of a human… Image Segmentation Pose Estimate??

  46. MRF for Interactive Image Segmentation, Boykov and Jolly [ICCV 2001] EnergyMRF = Unary likelihood Contrast Term Uniform Prior (Potts Model) Maximum-a-posteriori (MAP) solution x*= arg min E(x) x Data (D) Unary likelihood Pair-wise Terms MAP Solution

  47. However… • This energy formulation rarely provides realistic (target-like) results.

  48. Shape-Priors and Segmentation • Combine object detection with segmentation • Obj-Cut, Kumar et al., CVPR ’05 • Zhao and Davis, ICCV ’05 • Obj-Cut • Shape-Prior: Layered Pictorial Structure (LPS) • Learned exemplars for parts of the LPS model • Obtained impressive results = + Layer 1 Layer 2 LPS model

  49. LPS for Detection • Learning • Learnt automatically using a set of examples • Detection Tree of chamfers to detect parts, assemble with pictorial structure and belief propogation.

  50. Solve via Integer Programming • SDP formulation (Torr 2001, AI stats) • SOCP formulation (Kumar, Torr & Zisserman this conference) • LBP (Huttenlocher, many)

More Related