Efficient Inference for Fully-Connected CRFs with Stationarity

1. Efficient Inference for Fully-Connected CRFs with Stationarity Yimeng Zhang, Tsuhan Chen CVPR 2012

2. Summary • Explore object-class segmentation with fully-connected CRF models • Only restriction on pairwise terms is `spatial stationarity’ (i.e. depend on relative locations) • Show how efficient inference can be achieved by • Using a QP formulation • Using FFT to calculate gradients in complexity (linear in) O(NlogN)

3. Fully-connected CRF model • General pairwise CRF model: • Image I • Class labeling, X: • Label set, L: • V = set of pixels, N_i = neighbourhood of pixel i, Z(I) = partition function, psi = potential functions

4. Fully-connected CRF model • General pairwise CRF model: • In fully-connected CRF, for all i, N_i = V

5. Unary Potential • Unary potential generates a score for each object class per pixel (TextonBoost)

6. Pairwise Potential • Pairwise potential measures compatibility of the labels at each pair of pixels • Combines spatial and colour contrast factors

7. Pairwise Potential • Colour contrast: • Spatial term:

8. Pairwise Potential • Learning the spatial term

9. MAP inference using QP relaxation • Introduce a binary indicator variable for each pixel and label • MAP inference expressed as a quadratic integer program, and relaxed to give the QP

10. MAP inference using QP relaxation • QP relaxation has been proved to be tight in all cases (Ravikumar ICML 2006 [24]) • Moreover, it is convex whenever matrix of edge-weights is negative-definite • Additive bound for non-convex case • QP requires O(KN) variables, LP requires (K^2E)

11. MAP inference using QP relaxation • Gradient • Derive fixed-point update by forming Lagrangian and setting its derivative to 0

12. Illustration of QP updates

13. Efficiently evaluating the gradient • Required summation • Would be a convolution without the color term • With color term is requires 5D-filtering • Can be approximated by clustering into Ccolor clusters, => C convolutions across

14. Efficiently evaluating the gradient • Hence, for the case x_i = x_j, we need to evaluate • Instead, evaluate for C clusters (C = 10 to 15) • where • Finally, interpolate

15. Update complexity • FFTs of each spatial filters can be calculated in advance (K^2 filters) • At each update, we require C FFTs calculating, O(CNlogN) • K^2 convolutions are needed, each requiring a multiplication, O(K^2CN) • Terms can be added in Fourier domain, => only KC inverse FFTs needed, O(KCNlogN) • Run-time per iteration < 0.1s for 213x320 pixels (+ downsampling by factor of 5)

16. MSRC synthetic experiment • Unary terms randomized • Spatial distributions set to ground-truth

17. MSRC synthetic experiment • Running times

18. Sowerby synthetic experiment

19. MSRC full experiment • Use TextonBoost unary potentials • Compare with several other CRFs with same unaries • Grid only • Grid + P^N (Kohli, CVPR 2008) • Grid + P^N + Cooccurrence (Ladickỳ, ECCV 2010) • Fully-connected + Gaussian spatial (Krähenbühl, NIPS 2011)

20. MSRC full experiment • Qualitative comparison

21. MSRC full experiment • Quantitative comparison • Overall • Per-class • Timing: 2-8s per image