1 / 46

Learning to Segment from Diverse Data

Learning to Segment from Diverse Data. M. Pawan Kumar. Haithem Turki. Dan Preston. Daphne Koller. Learn accurate parameters for a segmentation model. Aim. Segmentation without generic foreground or background classes Train using both strongly and weakly supervised data.

toya
Download Presentation

Learning to Segment from Diverse Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Learning to Segment from Diverse Data M. Pawan Kumar Haithem Turki Dan Preston Daphne Koller

  2. Learn accurate parameters for a segmentation model Aim • Segmentation without generic foreground or background classes • Train using both strongly and weakly supervised data

  3. “Strong” Supervision “Weak” Supervision “Car” Data in Vision “One hand tied behind the back…. “

  4. Data for Vision “Strong” Supervision “Weak” Supervision  “Car”

  5. Specific foreground classes, generic background class Types of Data PASCAL VOC Segmentation Datasets

  6. Specific background classes, generic foreground class Types of Data Stanford Background Dataset

  7. Bounding boxes for objects Types of Data PASCAL VOC Detection Datasets Thousands of freely available images Current methods only use small, controlled datasets

  8. Image-level labels Types of Data ImageNet, Caltech … Thousands of freely available images “Car”

  9. Noisy data from web search Types of Data Google Image, Flickr, Picasa ….. Millions of freely available images

  10. Outline • Region-based Segmentation Model • Problem Formulation • Inference • Results

  11. Region-based Segmentation Model Regions Pixels Object Models 

  12. Outline • Region-based Segmentation Model • Problem Formulation • Inference • Results

  13. Region features Detection features Pairwise contrast Pairwise context Problem Formulation Treat missing information as latent variables Image x Annotation y Complete Annotation (y,h) Joint Feature Vector (x,y,h)

  14. Problem Formulation Treat missing information as latent variables Image x Annotation y Complete Annotation (y,h) Latent Structural SVM (y*,h*) = argmax wT (x,y,h) Trained by minimizing overlap loss ∆

  15. Self-Paced Learning hi = maxhH wtT(xi,yi,h) Update Update wt+1 by solving a biconvex problem min ||w||2 + C∑i vii - K∑i vi wT(xi,yi,hi) - wT(xi,y,h) ≥ (yi, y, h) - i Start with an initial estimate w0 Annotation Consistent Inference Loss Augmented Inference Kumar, Packer and Koller, 2010

  16. Outline • Region-based Segmentation Model • Problem Formulation • Inference • Results

  17. Generic Classes DICTIONARY OF REGIONS D MERGE AND INTERSECT WITH SEGMENTS TO FORM PUTATIVE REGIONS Current Regions Over-Segmentations ITERATE UNTIL CONVERGENCE SELECT REGIONS min Ty s.t. y  SELECT(D) Kumar and Koller, 2010

  18. Generic Classes Binary yr(0) = 1 iff r is not selected Binary yr(1) = 1 iff r is selected miny ∑r(i)yr(i) + ∑rs(i,j)yrs(i,j) Minimize the energy s.t. yr(0) + yr(1) = 1 Assign one label to r from L yrs(i,0) + yrs(i,1) = yr(i) Ensure yrs(i,j) = yr(i)ys(j) yrs(0,j) + yrs(1,j) = ys(j) ∑r “covers” u yr(1) = 1 Each super-pixel is covered by exactly one selected region yr(i), yrs(i,j)  {0,1} Binary variables

  19. Generic Classes DICTIONARY OF REGIONS D MERGE AND INTERSECT WITH SEGMENTS TO FORM PUTATIVE REGIONS Simultaneous region selection and labeling Current Regions Over-Segmentations ITERATE UNTIL CONVERGENCE SELECT REGIONS min Ty s.t. y  SELECT(D) ∆new ≤ ∆prev Kumar and Koller, 2010

  20. Examples Iteration 3 Iteration 6 Iteration 1

  21. Examples Iteration 3 Iteration 6 Iteration 1

  22. Examples Iteration 3 Iteration 6 Iteration 1

  23. Bounding Boxes Each row and each column of bounding box is covered min Ty y  SELECT(D) ∆new ≤ ∆prev +  Ka (1-za) za  {0,1} za ≤ r “covers” a yr(c)

  24. Examples Iteration 2 Iteration 4 Iteration 1

  25. Examples Iteration 2 Iteration 4 Iteration 1

  26. Examples Iteration 2 Iteration 4 Iteration 1

  27. Image-Level Labels Image must contain the specified object min Ty y  SELECT(D) ∆new ≤ ∆prev +  K (1-z) z  {0,1} z≤  yr(c)

  28. Outline • Region-based Segmentation Model • Problem Formulation • Inference • Results

  29. PASCAL VOC 2009 Stanford Background Dataset + Generic background class 20 foreground classes Generic foreground class 7 background classes

  30. PASCAL VOC 2009 Stanford Background Dataset + Train - 1274 images Validation - 225 images Test - 750 images Train - 572 images Validation - 53 images Test - 90 images Baseline: Closed-loop learning (CLL), Gould et al., 2009

  31. Results PASCAL VOC 2009 CLL - 24.7% LSVM - 26.9% Improvement over CLL SBD CLL - 53.1% LSVM - 54.3% Improvement over CLL

  32. PASCAL VOC 2009 + 2010 Stanford Background Dataset + Train - 1274 images Validation - 225 images Test - 750 images Bounding Boxes - 1564 images Train - 572 images Validation - 53 images Test - 90 images

  33. Results PASCAL VOC 2009 CLL - 24.7% LSVM - 26.9% BOX - 28.3% Improvement over CLL SBD CLL - 53.1% LSVM - 54.3% BOX - 54.8% Improvement over CLL

  34. PASCAL VOC 2009 + 2010 Stanford Background Dataset + Train - 1274 images Validation - 225 images Test - 750 images Bounding Boxes - 1564 images Train - 572 images Validation - 53 images Test - 90 images + 1000 image-level labels (ImageNet)

  35. Results PASCAL VOC 2009 CLL - 24.7% LSVM - 26.9% BOX - 28.3% LABEL - 28.8% Improvement over CLL SBD CLL - 53.1% LSVM - 54.3% BOX - 54.8% LABEL - 55.3% Improvement over CLL

  36. Examples

  37. Examples

  38. Failure Modes

  39. Examples

  40. Specific foreground classes, generic background class Types of Data PASCAL VOC Segmentation Datasets

  41. Specific background classes, generic foreground class Types of Data Stanford Background Dataset

  42. Bounding boxes for objects Types of Data PASCAL VOC Detection Datasets Thousands of freely available images

  43. Image-level labels Types of Data ImageNet, Caltech … Thousands of freely available images “Car”

  44. Noisy data from web search Types of Data Google Image, Flickr, Picasa ….. Millions of freely available images

  45. Two Problems The “Noise” Problem Self-Paced Learning The “Size” Problem Self-Paced Learning

  46. Questions?

More Related