Robust Higher Order Potentials For Enforcing Label Consistency

1. Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor Ladicky Philip Torr Oxford Brookes University, Oxford CVPR 2008

2. Image labelling Problems Assign a label to each image pixel Object Segmentation Image Denoising Geometry Estimation Sky Building Tree Grass

3. Object Segmentation using CRFs (Shotton et al. ECCV 2006) CRF Energy Unary potentials based on Colour, Location and Texture features Encourages label consistency in adjacent pixels

4. Limitations of Pairwise CRFs • Encourages short boundaries (Shrinkage bias) • Can only enforce label consistency in adjacent pixels • Inability to incorporate region based features Image Unary Potential MAP-CRF Solution

5. Label Consistency in Image Regions • Pixels constituting some regions belong to • Same plane (Orientation) • (Hoiem, Efros, & Herbert, ICCV’05) • Same object • (Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06) Image (MSRC) Segmentation (Mean shift)

6. Image labelling using segments Unsupervised Segmentation Object Labelling Image • Geometric Context • [Hoiem et al, ICCV05] • Object Segmentation • [He et al. ECCV06, Yang et al. CVPR07, Rabinovich et al. ICCV07, Batra et al. CVPR08] • Interactive Video Segmentation • [Wang, SIGGRAPH 2005 ] Not robust to Inconsistent Segments!

7. Our Higher Order CRF Model Encourages label consistency in regions Multiple Segmentations c

8. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c

9. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: 0

10. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: f (|c|)

11. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] Does not distinguish between Good/Bad Segments ! c Cost: f (|c|)

12. Quality based Label Consistency Label inconsistency cost depends on segment quality • How to measure quality G(c)? • [Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others] • Colour and Texture Similarity • Contour Energy Measure quality from variance in feature responses Higher order generalization of contrast-sensitive pairwise potential

13. Quality based Label Consistency Segment Quality (darker is better) Mean shift segmentation MSRC image

14. Robust Consistency Potentials gmax PN Potts 0 Too Rigid! 0 1 Inconsistent Pixels gmax 0 Robust 0 1 T Inconsistent Pixels

15. Robust Consistency Potentials Maximum Inconsistency Cost Number of Inconsistent Pixels Slope gmax 0 Robust 0 1 T Inconsistent Pixels

16. Minimizing Higher order Energy Functions • Message passing is computationally expensive • High runtime and space complexity - O(LN) • L = Number of Labels, N = Size of Clique • Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007] • 2x2 clique potentials for Image Denoising • Take minutes per iteration (Hours to converge)

17. Minimizing Higher order Energy Functions • Graph Cut based move making algorithm [Kohli et al. CVPR 2007] • Can handle very high order energy functions • Extremely efficient: computation time in the order of seconds • Only applicable to some classes of functions (PN Potts) • Cannot handle robust consistency potential • This paper • Can minimize a much larger class of higher order energy functions • Same time complexity as [Kohli et al. CVPR 2007]

18. Move making algorithms Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] • Makes a series of changes to the solution (moves) • Each move results in a solution with smaller energy Current Solution How to minimize move functions? Move to new solution Generate pseudo-boolean move function Minimize move function to get optimal move

19. Minimizing Move Functions using Graph Cuts Most pairwise CRF models used in Computer Vision lead to submodular move functions Second order Pseudo-boolean Function Minimization (submodular) st-mincut (Positive weights) Optimal moves can be found extremely efficiently using graphs cuts

20. Minimizing Higher Order Energy Functions: Our results • Result 1 • We show that a large class of higher order potentials lead to higher order submodular move functions • Can be minimized in polynomial time • Submodular Function Minimization • Minimizing general submodular functions is computationally expensive • Complexity O(n6) • Cannot handle large problems! Details in Technical Report

21. Minimizing Higher Order Energy Functions: Our results • Minimizing Higher order functions using Graph cuts • Higher order functions can be transformed to second order functions by adding auxillary variables • Exponential number of auxillary variables needed in general • Result 2 • Our higher order functions can be transformed to second-order functions using ≤2 auxillary variables per potential. • Can be minimized extremely efficiently • Complexity << O(n6) Details in Technical Report

22. Overview of our Method Higher Order Energy Unary Potentials [Shotton et al. ECCV 2006] + Energy Minimization Contrast Sensitive Pairwise Potentials + Segmentation Solution Higher Order Potentials (Multiple Segmentations)

23. Experimental results Datasets: MSRC (21), Sowerby (7) [Shotton et al. ECCV 2006] [He et al. CVPR 04]

24. Qualitative Results Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth Grass Sheep

25. Qualitative Results (Contd..) Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV 2006

26. Quantitative Results: Problems Rough ground truth segmentations Fine structures have small influence on overall pixel accuracy

27. Generating Accurate Segmentations • Generated accurate segmentation of 27 images • 30 minutes per image Image (MSRC-21) Original Segmentation New Segmentation

28. Relationship between Qualitative and Quantitative Results Pairwise CRF Higher order CRF Ground Truth Image (MSRC-21) Overall Pixel Accuracy 95.8% 98.7% Small changes in pixel accuracy can lead to large improvements in segmentation results.

29. Quantitative Accuracy • Measure accuracy in labelling boundary pixels. • Accuracy evaluated in boundary bands of variable width Trimap (8-pixels) Trimap (16-pixels) Hand-labelled Segmentation Image (MSRC-21)

30. Quantitative Accuracy • Measure accuracy in labelling boundary pixels. • Accuracy evaluated in boundary bands of variable width

31. Conclusions • Method to enforce label consistency in image regions • Generalization of the commonly used Pairwise CRF model • Allows integration of pixel and region level features for image labelling problems

32. Thanks

33. Running Time Results Time (sec) Number of Segmentations Inconsistency Cost

34. Qualitative Results (Contd..) Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth

35. Transformation to second order functions c Auxiliary variables