1 / 58

Speeding Up MRF Optimization using Graph Cuts for Computer Vision

Speeding Up MRF Optimization using Graph Cuts for Computer Vision. Vibhav Vineet Adviser: Prof. P. J. Narayanan. Labelling Problem. Extracting Foreground Pixels. Disparity map calculation. Image Denoising. Extracting Foreground Object. Left Tsukuba Image. Noisy House Image. Disparity Map.

nicolecannon
Download Presentation

Speeding Up MRF Optimization using Graph Cuts for Computer Vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Speeding Up MRF Optimization using Graph Cuts for Computer Vision Vibhav Vineet Adviser: Prof. P. J. Narayanan

  2. Labelling Problem Extracting Foreground Pixels Disparity map calculation Image Denoising Extracting Foreground Object Left Tsukuba Image Noisy House Image Disparity Map Denoised Image Flower Image Pixel Labeling: Assigning a label to each pixel in image. Pixel Labeling: Assigning a label to each pixel in image. Pixel Labeling: Assigning a label to each pixel in image. Image Segmentation: Involves Separating Foreground layer from background layer Stereo Correspondence: Involves Calculation of depth map using left and right images Image Denoising: Involves assigning denoised intensity value to each pixel in image.

  3. Labelling Problem • - MAP Estimation • To find the best possible configuration. • But the complexity increases • With the number of variables/pixels • With the number of labels in the label set • Using joint probability or conditional probabilities to evaluate the best possible configuration • Very hard with the limited computation and memory power • Energy minimization method • MAP – MRF equiivalence • Methods provide approximate solution at a moderate times • Generally, in computer vision an energy function involve unary cost and pairwise interactions between variables.

  4. Labelling Problem Image-Graph Equivalence Image-Graph Equivalence Image-Graph Equivalence Graph G( V, E ) Unary Cost ( Per Vertex Cost) Total Cost = Unary Cost + PairWise Cost Different Labeling, different Cost Energy E ( X ) = Unary Potential + PairWise Potential Graph G( V, E ) PairWise Cost ( Per Edge Cost ) Cost of Assignment for “fg” is low. Cost of Assignment for “bg” is high. Cost of Assignment for “same label” is low. Cost of Assignment for “different labels” is high. Labeling Problem: Find a Labeling ( X ) with Minimum Cost or Energy Value

  5. MAP-MRF Formulation MAP(X) Min Energy(X*) MAP estimation of a configuration X is equivalent to the minimum energy defined over the configuration Energy = Data Term + Smoothness Term Graph Cuts in Computer Vision Image Energy Function Graph Construction st-MinCut

  6. Graph Construction Vertex per pixel s Image-Graph Equivalence Background Pixels Foreground Pixels Image Graph G(V,E) Add n-edges Add t-edges • Graph Constructed for vision problems • Grid graphs • Low connectivity • Connectivity is limited to 4, 8, or 27 t

  7. The st-Mincut Problem • Given a Graph G(V,E,W) and two vertices s and t. • Partition G into two disjoint components containing s and t respectively such that sum of edge weights from s to t is minimum s Mincut t

  8. Computing the st-Mincut • Solve the dual Maximum Flow problem • Two approaches • Edmond Karp’s Augmenting path method • Goldberg’s Push-Relabel method st-Mincut Max Flow Dual In every network, the maximum flow equals the cost of st-mincut

  9. Edmond Karp Method • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t s s s s s s s 93 100 100 93 100 87 100 37 37 40 37 40 37 37 6 6 6 6 6 6 0 0 6 3 3 7 7 7 0 7 7 7 7 3 3 3 3 3 13 13 13 7 13 13 13 17 11 11 11 11 11 11 14 14 14 14 1 1 1 1 1 7 7 1 4 4 Current Flow: 10 Current Flow: 0 Current Flow: 3 Current Flow: 10 Current Flow: 0 Current Flow: 3 Current Flow: 16 t t t t t t t • Flow <= Edge Capacity • Edge Capcity must be positive

  10. Edmond Karp Method • Initialize flow in G to 0 • Find a shortest path from s to t. • Augment the path with minimum possible flow • Repeat until there exists a path from s to t s 87 37 6 7 3 7 17 1 1 Current Flow: 16 t • Flow <= Edge Capacity • Edge Capcity must be positive

  11. Goldberg’s Push-Relabel Algorithm • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation s s s s s s s s s s s s s s s Height Height Height Height Height Height Height Height Height Height Height Height Height Height Height 40 100 0 0 0 7 7 0 13 6 6 13 6 6 13 13 13 13 100 87 87 100 37 40 37 37 37 37 37 37 37 37 40 37 37 37 87 87 87 87 87 87 87 87 87 87 7 7 7 7 6 7 0 0 0 6 7 7 0 0 0 0 3 0 0 3 0 0 0 0 7 7 7 7 3 3 7 13 13 13 13 13 3 7 7 7 7 7 7 7 7 7 7 11 17 11 17 17 11 11 17 11 11 3 3 3 3 3 3 3 3 3 3 100 7 7 0 0 0 6 0 0 6 0 t t t t t t t t t t t t t t t 1 1 1 1 1 1 1 4 4 1 0 3 0 3 0 3 3 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 8 8 8 8 8 1 8 14 14 6 6 6 6 6 6 6 6 6 6 11 11 11 11 11 4 4 4 4 4 14 14 14 14 14 6 6 6 6 Current Flow: 0 Current Flow: 16 Current Flow: 0 Current Flow: 0 Current Flow: 9 Current Flow: 0 Current Flow: 9 Current Flow: 9 Current Flow: 0 Current Flow: 9 Current Flow: 9 Current Flow: 0 Current Flow: 9 Current Flow: 9 Current Flow: 0 6 • Flow <= Edge Capacity • Edge Capcity must be positive

  12. Goldberg’s Push-Relabel Algorithm • Initialize excess flow and heights in G • Perform an applicable Push or Relabel operation • Repeat until there exists an applicable push or relabel operation s Height 0 6 37 87 0 0 7 17 3 t 1 0 1 6 Current Flow: 16 • Flow <= Edge Capacity • Edge Capcity must be positive

  13. Motivation • Fast Computation Required • Robot navigation, surveillance, video processing etc • Video Processing at real time • You tube and other web-servers • Large images Processing • Even our offshelf cameras take high resolution images • Interactive tools

  14. Mapping to CUDA Image Grid Thread per pixel Image CUDA Grid CUDA Block

  15. Push-Relabel Algorithm on CUDA • Push is an local operation with each node sending flows to its neighbors. • Relabel is also a local operation, each vertex updates its own height. • Problems faced: • Read After Write consistency • Synchronization of threads

  16. Handling Problems using Atomics • Push operations can performed without any read after write inconsistencies • Relabel is a per vertex operation • Employing atomic Capabilities and combining the push and pull kernels • Push Kernel • Relabel Kernel • Lowers Global memory access, empirically faster convergence is observed.

  17. The Push Kernel • Load heights from the global memory to the shared memory. • Synchronize threads ensuring the completion of load operation. • Push flows to eligible neighbors atomically. • Update the edge-weights atomically in the residual graph. • Update excess flow atomically in the residual graph.

  18. The Relabel Kernel • Load height from the global memory to the shared memory. • Synchronize ensuring the completion of load operation. • Compute the minimum height of all neighbors and set own height to plus one of this. • Write the new height to global memory.

  19. Using Shared Memory • Per CUDA block requires (Block_size+2 X Block_size+2) memory to be loaded into the shared memory • Corner pixels need heights from other blocks Block size + 2 Thread Block size + 2 Height needed per thread Block

  20. Heuristics on Push and Relabel • On grid graphs Global relabel (BFS based) is an expensive operation • Local relabel perform better empirically • Multiple pushes can be performed before applying a relabel step using • For most general graphs m=3 and k=7 are found to be optimal. (m*Push + Relabel)*k + Global Relabel

  21. Stochastic Cuts • MRF consists of simple and difficult pixels. • Simple pixels get their correct labels in few initial iterations • Difficult (few) pixels exchange flows with their neighbors in later iterations • Stochastic Cuts processes pixels based on their activity • Activity is based on change in flows from previous to current iteration. Low activity is observed for simple pixels • Heuristically process simple pixels after a fixed number of iterations

  22. Experimental Results

  23. Experimental Results

  24. Graph Reparameterization S 2 9 1 2 5 4 t S 2 9 Graph Cuts Graph Cuts 1 2 5 4 t

  25. Graph Reparameterization S 2 9 1 2 5 4 t S 2 9 Graph Cuts Graph Cuts 1 2 5 4 t Graph Reparameterized Graph Reparameterized No change in cut S S 2+2 4 9 9 1 Graph Cuts Graph Cuts 1 2 2 5+2 4 7 4 t t

  26. Dynamic Cuts EA SA EB SB Problem Instance 1 Problem Instance 2 Problems instances where they differ slightly. Solving each independently is computationally expensive Example: Continuous frames in a video

  27. Dynamic Cuts Steps Involved • Edge capacities are updated and reparameterized using • Previous frame edge capacities • Previous frame residual flow • Current frame edge capacities ci Previous Frame fi Previous Frame after st-MinCut ri Current Frame ci’ Updation Step: ri’ = ri + ci’ - ci ri’ Approximate cut using previous frame and its st-MinCut fi’ Reparameterization Step: rsi’ = 0 rit’ = cit - fit + fsi – csi’ Final st-MinCut of current frame

  28. Dynamic Cuts are parallizable • Updation and Reparameterization are independent and parallizable operations, work locally at every vertex. • st-Mincut is performed using a parallel implementation of Push Relabel algorithm.

  29. Dynamic Cuts Empirically • Running time depends on the percentage of weights that changed • On a low resolution video, the dynamic cuts takes about 2 ms compared to 7 ms on the same image for the st-MinCut Consecutive frames of a video segmented using dynamic cuts

  30. The Multilabeling problem • Multi-way cut on any graph is an NP-Hard problem for L > 2 • Approximate solutions based on graph cuts • α-Expansion • αβ-Swap

  31. The α-Expansion 1: Initialize the MRF with an arbitrary labeling X 2: For each label alpha \in L do 3: Construct the graph based on the current configuration 4: Perform one α-Expansion step (st-cut) 5: Update the configuration if energy decreases 6: End For 7: Repeat steps 2 to 6 till convergence. Step 2-6 is a cycle and 3-5 is an iteration

  32. Incremental α-Expansion • Reusability of flows, as in dynamic MRF • Better initializations for next graph cut • Incremental/Dynamic • Reuse the flows from label to label and and re-cycle flows from cycle to cycle. Cycle 1 Label1 Label2 Input Label3 Cycle 2

  33. Incremental α-Expansion Results Tsukuba Teddy Panorama Penguin

  34. Incremental α-Expansion Results Total Timings on Different Datasets

  35. Processing on High Detailed Scene • High Detailed Scene • High Resolution Image • High Dynamic Ranges of Colors • Wide View Angles. • Challenges • High Computation Cost • High Memory Requirement • Interaction with high resolution images • Statistics of image sizes available on Google images. • An overwhelming fraction of images are of size 2 to 10 million pixels. • Only 0.6% of fewer images had more than 40 mega pixels.

  36. Processing on High Detailed Scene Define E(x) for coarsest image Final Result for this level Define E(x) for next finer level Final Result at this level Solve an optimization problem at the coarser level to dynamically update the optimization instance for the next level for better initialization.

  37. Pyramid Reparameterization Pyramid Construction Pyramid is Constructed. Input largest image at the base of the pyramid. Each pixel coarser level image is mean of 4 pixels at the previous finer level

  38. Pyramid Reparameterization Minimization at (i-1) Image (i-1) Graph G(V,E) Segmented Image at (i-1) Residual Graph G(V,E) Upsampling Step Upsampled Initial Graph Upsampled Residual Graph

  39. Pyramid Reparameterization Computationally Expensive Graph Cuts Graph at the current level Final Residual Graph at the current level

  40. Pyramid Reparameterization Graph Cuts Upsampled graph of previous level Reuse of flows Difference between two graphs Cheaper solution Computationally Expensive Graph Cuts Graph at the current level Final Residual Graph at the current level

  41. Upsampling Rules Graph Upsampling Graph Cuts Residual Graph Upsampling

  42. Upsampling Rules Graph Upsampling Graph Cuts Residual Graph Upsampling

  43. Upsampling Rules Graph Upsampling Graph Cuts Residual Graph Upsampling

  44. Image Segmentation Results Horse 3.3 MP (2048x1600)

  45. Image Segmentation Results

  46. Interactive Image Segmentation Tool • User Interaction Important in foreground/background separation User 1 User 2 • Results of a user study on image size for comfortable manipulation for two display sizes. • Average subjective response for six image sizes. • Images that are larger than the display is disfavored users.

  47. Pyramid Segmentation System Display Window • User interacts at the display window of comfortable size Actual Image • Actual Segmentation goes on in background on other levels • Quick Segment: • Display the segmentation results on this display image • Provides perceptual response to start planning further interactions

  48. User Study Results of User Study on CPU and GPU version of Pyramid Segmentation With GrabCut and Quick Selection Interaction Time Response Time Total Time Subjective Response of Users

  49. Multiresolution alpha-expansion • - Build pyramid of graphs. • Perform alpha-expansion at a lower resolution graph. • Save the initial and final residual graphs for all the labels. • Upsample and reparameterize the previous resolution initial and final graphs and current resolution initial graph. • Perform alpha-expansion at this level. • Repeat this for all the levels in the pyramid.

  50. Stereo Correspondence Image size – 1328 x 1104 Number of Labels (Disparity) – 200 - 290

More Related