Efficiently Selecting Regions for Scene Understanding

1. r1 r4 r5 r2 r3 r6 Efficiently Selecting Regions for Scene Understanding S T A N F O R D Aim:To efficiently select accurate, discriminative regions for a high-level vision task Results Integer Program Semantic Segmentation Integer constraints yr(i)  {0,1} Background Dataset - 715 images S - set of super-pixels (intersection of segments) 80/20 train/test split, 4 folds Over-segmentations as Regions Minimize Energy Simple Inference r,r’  D minyr,i r(i)yr(i) + (r,r’),i,j rr’(i,j)yrr’(i,j) Baselines • Pixel-based Model Uniqueness i yr(i) = 1 i  L’ = {0} (not selected)  L SEGMENTS SEGMENTS RESULT RESULT • Intersection of over-segmentations j  L’, (r,r’) - neighbors Marginalization j yrr’(i,j) = yr(i) • Lowest energy over-segmentation r “covers” s iL yr(i) = 1 Each super-pixel is covered Covering • Gould et al., 2009 I M A G E Linear Programming Relaxation Relax yr(i)  [0,1] SCENE LAYOUT, Hoiem et al, 2005 MONOCULAR 3D, Saxena et al, 2008 Standard pairwise energy. Dual decomposition with tree slaves. • Segments are not accurate • Do not align with scene boundaries SEGMENTS RESULT Sub-dictionary DT D that forms a tree. Union of DT = D. P I X E L Minimize energy such that uniqueness, marginalization and integer constraints are satisfied. • Segments are not discriminative • Too small to capture useful cues r1 r5 Belief propagation r2 r3 r4 SEGMENTATION, Gould et al, 2008 I N T E R Overview Region Selection using Energy Minimization Tree slaves do not enforce the covering constraint. Large number of possible pixel-to-region assignments Sub-dictionary DC D that covers s. One slave for each s  S. S E G M Exact inference is intractable Move-making algorithm s Minimize energy such that exactly one region in DC is selected. Linear Search Over Regions DICTIONARY OF REGIONS D MERGE AND INTERSECT WITH SEGMENTS TO FORM PUTATIVE REGIONS r1 r3 r2 G O U L D Standard linear programming relaxation is not tight. Clique constraints Choose a subset of 3 super-pixels SQ  S. CURRENT REGIONS SEGMENTATIONS Sub-dictionary DQ D, each region covers at least one s  SQ SELECT REGIONS ITERATE UNTIL CONVERGENCE O U R Minimize energy such that uniqueness, marginalization, covering and integer constraints are satisfied. Efficient Enumeration Select regions from D (with overlapping regions) such that • No regions overlap • Entire image is covered Integer Program • Energy of the vision task is minimized Shrinking Strategy Maintain a Small Active Set of Regions De-activate regions not selected in any slave for T iterations Two dictionary moves • Merge neighboring regions • Merge and intersect regions with segments Iteratively introduce cliques with maximum active regions Statistically significant improvement