Tiered Scene Labeling

1. Tiered Scene Labeling Pedro Felzenszwalb University of Chicago Olga Veksler University of Western Ontario

2. Global Optimum • Global optimization for 2D labeling is very rare • Tiered Labeling is a new 2D instance where global minimum can be found • Global minimum can make a difference between success and failure • Example: segmentation with “bow” shape prior input image: local minimum with expansion algorithm [Boykov et.al. PAMI’01]: global minima with our method:

3. Pixel Labeling Problems image I labels L ={ } • Assign a label from L to each pixel in image I • Applications: Optical Flow Lempitskyet.al. CVPR08 Segmentation Delonget.al. CVPR10 Stereo Boykov et.al. PAMI01

4. Energy Function bad labeling good labeling fq fp label preferences • Find labeling f minimizing energy: • Special cases optimized exactly • Ishikawa’03, Schlesinger et.al.’06, Kolmogorov et.al.’07 • Most formulations are NP-hard • for example, Potts Vpq(fp,fq)=min(1,|fp-fq|) • approximations such as expansion [Boykov et.al. PAMI01]

5. Restricting Label Layout Liu, Veksler, Samarabandu [PAMI10] five labels with ordering constraints T L C R B • Standard methods do not work well • expansion algorithm gets stuck to a bad local minima • although optimizing not NP-hard for Potts model • their exact algorithm not very practical

6. Tiered Labeling Two curves to separate into top, bottom and middle Plus vertical boundaries in the middle a a b c d c • Main idea: for certain restricted label layouts, we can optimize efficiently and exactly T B • Main technical part: fast dynamic programming for exact optimization, for arbitraryVpq

7. Application: Geometric Labeling (Hoiem et.al.) Label T Sky Label B ground Labels L, C, R surface facing left, center, right sky L C R ground data terms from Hoiem our result input

8. Application: Shape Prior for Binary Segmentation Prior encourages certain orientations between object parts and background labels input image “plane” prior “leaf” prior “bow” prior “cross” prior

9. Dynamic Programming for 1D U( ) U( ) U( ) • 1D labeling problem • pixels are usually called sites, labels are usually called states s0 s1 sk sm-1 U( ) U( ) H(s0,s1) … … • Complexity for all sites is O(p2m) • m is number of site, p is number of states

10. Tiered Labeling  DP: Overview • Collapse 2D structure to 1D structure i j n-1 … … s0 s1 sk sm-1 • A labeling in column k corresponds to a state of sk • State space for sk does not “explode” due to tiered structure of a labeling

11. Tiered labeling: State Space T T i l l j B B n-1 sk • Labeling for column k defined by (i,j,l) • i: boundary of the top and middle regions • j: boundary of the middle and bottom regions • l: label of the middle region • O(n2p) states for image with n rows and p labels

12. Tiered Labeling DP: Unary Terms data terms of the labeling problem inside column pairwise terms of the labeling problem inside column T T l l B B sk U(sk) = D( )+D( )+D( )+D( )+D( )+ l B T T T T B B B B l l l l l + V( ,, )+V( ,, )+V( ,, )+V( ,, )+V( ,, ) • Each is computed in O(1) time • with pre-computed cumulative sums

13. Tiered Labeling DP: Pairwise Terms T T V( , ) l' T + V( , ) l' l + V( , ) pairwise terms of labeling problem between columns B l + V( , ) B B + V( , ) B B + V( , ) T T H(sk,sk-1) = l' T l' l l B B B B B sk-1 sk • Each is computed in O(1) time • with pre-computed cumulative sums

14. Tiered Labeling DP: Optimization m • n by m image, p labels • m sites, n2p states each • Classical DP • O(mn4p2) – very slow • Speedup using structure of the problem • O(mn2p2) • O(mn2p) for Potts model n-1 s0 s1 … … sk sm-1 • about a second for reasonable size problem

15. Tiered Labeling DP: Speed Ups • Basic step of DP: • for given (i, j) in column k, find the best (i’, j’) in the previous column • brute force search is O(n2p) T T i l T i' l l' j l' B j' • We find the best (i’, j’) for all (i, j) simultaneously in O(n2p) time • Speedup #1: • decouple search for i’ and j’ • saves O(n) factor • Speedup #2: • many almost identical min-sums computations • saves another O(n) factor B B B B sk-1 sk

16. Geometric Labeling Results Soft Prior encourages sequence L,C,R certain orientation between L and ground, etc. Hoiem et.al. accuracy 78.1% Our results: accuracy 81.4% Hoiem et.al. sky C R L Ours ground failure due to tiered layout violation

17. Shape Prior for Binary Segmentation small penalty small penalty medium penalty heavy penalty heavy penalty huge penalty T T B B B T T B T T B B • Background with 2 parts: T and B • Object with 5 parts: T B B T

18. Interactive Segmentation • Generic “plane” prior • 4 parts for object • Generic enough to capture significant variation in shape • Ensure the “middle” regions forms one connected component • Standard methods, like expansion gets stuck in a bad local minima immediately no prior with prior

19. Shape Prior for Binary Segmentation • Unary terms are mixture of Gaussians • Different shape priors no prior prior

20. Summary • Considering restricted labelling layout leads to efficient and exact DP optimization • handle arbitrary Vpq • Limitations • tiered labelling model is restrictive • 4-connected neighborhood system • Future directions • other interesting “tiered” models • more than 1 label in T and B regions • unary terms can be more general • depend on a whole column at a time, i.e. a very high order clique • learn shape priors from labelled examples