Image Segmentation with A Bounding Box Prior

1. Image Segmentation with A Bounding Box Prior International Conference on Computer Vision 2009 (ICCV 2009) ORAL Victor Lempisky, PushmeetKohli, CarstenRother, Toby Sharp Microsoft Research Cambridge

2. Introduction • Restrict the attention of segmentation process to the interior of the bounding box

3. Introduction Need to use high level information to do better But “tightness” constraint would help! The desired segmentation should have parts that are sufficiently close to each of the sides of the bounding box

4. Introduction Goal: develop a new segmentation frame work that is capable of enforcing such tightness

5. Key aspects and contributions • Proposes a formalization of the notion of tightness • Write down as an integer program (IP) with incorporating low-level cues • Investigate techniques for the approximate solution of the IP • Convex continuous relaxation • Pinpointing algorithm • Evaluate the proposed algorithm on a publicly-available dataset of segmentation problem with ground truth • Show that there are good results corresponding to previous papers

6. Problem Formulation • Assume that image corresponds to the subimage inside the user-specified bounding box s • Graph cut segmentation [Boykov & Jolly 2001] integrates cues and input via: p q ... ... ? t Minimize subject to “shape is sufficiently tight”

7. Tackling Tightness To formalize this “sufficient tightness”, introduce the notion of margins Upper margin Tightness (Predefined) : the height of top and bottom and the width of left and right (d=0.06) crossing path Left margin Right margin Introduce the tightness via the concept of crossing path Middle box crossing path Crossing path: the path inside the middle box and endpoints on the opposite direction Lower margin

8. Tackling Tightness • There are the two types of crossing paths • Short crossing path (Short CP) • Long crossing path (Long CP) • Inside the middle box • Endpoints on the opposite sides Top margin • Inside the middle box • Endpoints on the left and right sides • (top) (bottom) • goes between the top and bottom margins • (right) (left) Short CP Left margin Right margin Middle box Long CP Bottom margin

9. Shapes Tightness • Shape tightness: intersect all crossing paths, any one of two types or both types Tight: Not tight:

10. Shapes Tightness • The are two classes of the shapes tightness • Strong tight intersect all short crossing paths • Weakly tight intersect only all long crossing paths Strong tight Corollary: the shape is tight if and only ifthe shape has a connected component touching all 4 sides of the middle box Weakly tight Corollary: When the thickness of all four margins is increased, the new set of weakly tight shapes is a superset of old one

11. Incorporating Shape Tightness bounding box middle box C2 the set of all crossing paths C1

12. Optimize the Problem Formulation • Need some approximations because of NP-hard problem • It is a IP non-convex • The number of constraints in (c) is combinatorial Solved by an iterative method Relax and transform it into LP

13. Relaxation Algorithm In the first iteration, After the LP from the previous iteration is solved, pick a group of crossing paths from violating the constraints by some small tolerance and add the group to Solve the LP with the new set Termination: until constraints of all crossing paths are satisfied ( within tolerance ) and further constraints may be added in step 2 • Each time, consider only a subset of all crossing paths in the constraint set

14. How to Select?? • : the solution in the previous iteration • Continue picking the paths with the smallest sum that do not intersect with the previous paths () • Each time the optimal path found by two runs of Dijkstraalgorithm ( from top to bottom and from left to right) • Picked at most at a time, where is the largest dimension of and is the margin thickness e.g. the width and length of the middle box • Stop until the sum exceeds • All the picked constraints are added to the set

15. How to Round?? • The solution of LP is a decimal fraction • Need to be rounded to the integer solution • The simplest and popular method is to use the threshold However, the selection of threshold is very important to ensure that the rounded solution satisfies all crossing paths constraints • Present a new algorithm called pinpointing which is better than the threshold-based rounding

16. Pinpointing Algorithm • Idea: build a prior map by using the solution of LP and use the prior map to guide the construction of the pinpointing set • Based on graph cut minimization and approximates the IP with another IP where is a pointing set that contains the hard assigned pixels • There are two stages: • During the first stages, greedily expanded according priority • During the second stages, greedily removed

17. Pinpointing Algorithm Stages Second stage bounding box bounding box • First stage middle box middle box middle box 1 0 LP solution (priority map) Rounded solution step Empty set The hard assigned pixels Use “dynamic graph cut” [Boykov & Jolly,2001],[Kohli & Torr, 2004] Corollary: pinpointing gives lower energy than threshold

18. Experiment Error rate • Set the thickness of all four margins to d=0.06 of the largest bounding box dimension • Use the grabcut dataset Thickness

19. Experiments • Pairwise and unary terms • Pairwise term: • Use the 8-connected edge set • Enclidean norm • : the RGB color vectors • and

20. Experiments • Unary term: • Fit the Gaussian mixture model to the color vectors • : the background GMM to the 33% pixels with smallest probability • : the foreground GMM to the 33% pixels with largest probability • Set the unary terms of all pixels along the bound sides to

21. Relative Performance (1) (2) (3) (4) (5) (6) (1): whererver (2): Use the graph cut without the tightness (3): Solve the IP by solving the LP and rounding using thrsholding (4): Solve the IP by solving the LP and rounding using pinpointing (5): Solve the IP by pinpointing with priority map based on unary terms ( P) (6): Solve the IP by pointing with minus 1-Monmarginals [Kohli & Torr, 2005] as the prior map

22. Relative Performance

23. Iterative Process • GrabCut-GC: a conventional GrabCut algorithm that uses standard graph cut minimization for all segmentation steps • GrabCut-Pinpointing: a new algorithm that enforces the tightness prior on each step. For the sake of speed, the Unary-Pinpointing algorithm was used to solve the IP • Two way to initialized mixture models: • InitThirds: as used in the above experiment • InitFullBox: as suggested in [Rother et al., 2004]

24. Iterative Process