Self-Validated Labeling of MRFs for Image Segmentation

1. Self-Validated Labeling of MRFs for Image Segmentation Accepted by IEEE TPAMI Wei Feng1,2,Jiaya Jia2and Zhi-Qiang Liu1 1. School of Creative Media, City University of Hong Kong 2. Dept. of CSE, The Chinese University of Hong Kong

2. Outline • Motivation • Graph formulation of MRF labeling • Graduated graph cuts • Experimental results • Conclusion

3. Outline • Motivation • Graph formulation of MRF labeling • Graduated graph cuts • Experimental results • Conclusion

4. Self-Validated Labeling • Common problem: segmentation, stereo etc. • Self-validated labeling: two parts • Labeling quality: accuracy (i.e., likelihood) and spatial coherence • Labeling cost (i.e., the number of labels) • Bayesian framework: to minimize the Gibbs energy (equivalent form of MAP)

5. Motivation • Computational complexity remains a major weakness of the MRF/MAP scheme • Robustness to noise • Preservation of soft boundaries • Insensitive to initialization

6. Motivation • Self-validation: How to determine the number of clusters? • To segment a large number of images • Global optimization based methods are robust, but most are not self-validated • Split-and-merge methods are self-validated, but vulnerable to noise

7. Motivation • For a noisy image consisting of 5 segments • Let’s see the performance of the state-of-the art methods

8. Motivation • Normalized cut (NCut)[1] • Unself-validated segmentation (i.e., the user needs to indicated the number of segments, bad) • Robust to noise (good) • Average time: 11.38s (fast, good) • NCut is unable to return satisfying result when feeded by the right number of segments 5; it can produce all “right” boundaries, mixed with many “wrong” boundaries, only when feeded by a much larger number of segments 20. [1] J. Shi and J. Malik, “Normalized cuts and image segmentation”, PAMI 2000.

9. Motivation • Bottom-up methods • E.g., Mean shift [2] • E.g., GBS [3] • Self-validated (good) • Very fast (<1s, good) • But, sensitive to noise (bad) [2] D. Comaniciu and P. Meer. “Mean shift: A robust approach towards feature space analysis”, PAMI 2002. [3] P. F. Felzenszwalb and D. P. Huttenlocher. “Efficient graph based image segmentation”, IJCV 2004.

10. Motivation • Data-driven MCMC[4] • Self-validated (good) • Robust to noise (good) • But, very slow (bad) [4] Z. Tu and S.-C. Zhu, “Image segmentation by data-driven Markov chain Monte Carlo”, PAMI 2002.

11. Motivation • As a result, we need a self-validated segmentation method, which is fast and robust to noise. • Our method: graduated graph mincut • Tree-structured graph cuts (TSGC) • Net-structured graph cuts (NSGC) • Hierarchical graph cuts (HGC)

12. Motivation [5] [5] C. D’Elia, G. Poggi, and G. Scarpa, “A tree-structured Markov random field model for Bayesian image segmentation,” IEEE Trans. Image Processing, vol. 12, no. 10, pp. 1250–1264, 2003.

13. Outline • Motivation • Graph formulation of MRF labeling • Graduated graph cuts • Experimental results • Conclusion

14. Graph Formulation of MRFs • Graph formulation of MRFs (with second order neighborhood system N2): (a) graph G = <V,E> with K segments {L1, L2 . . . LK } and observation Y; (b) final labeling corresponds to a multiway cut of the graph G.

15. Graph Formulation of MRFs • Property: Gibbs energy of segmentation Seg(I) can be defined as • MRF-based segmentation ↔ multiway (K-way) graph mincut problem (NP-complete, K=2 solvable)

16. Outline • Motivation • Graph formulation of MRF labeling • Graduated graph cuts • Experimental results • Conclusion

17. Graduated Graph Mincut • Main idea • To gradually adjust the optimal labeling according to the Gibbs energy minimization principle. • A vertical extension of binary graph mincut (in constrast to horizontal extension, α-expansion and α-β swap)

18. Graduated Graph Mincut

19. Binary Labeling of MRFs

20. Binary Labeling of MRFs

21. Tree-structured Graph Cuts

22. Tree-structured Graph Cuts

23. Tree-structured Graph Cuts : (over-segmentation)

24. Net-structured Graph Cuts

25. Net-structured Graph Cuts

26. Net-structured Graph Cuts

27. Hierarchical Graph Cuts

28. Hierarchical Graph Cuts

29. Graduated Graph Cuts • Summary • An effective tool for self-validated labeling problems in low level vision. • An efficient energy minimization scheme by graph cuts. • Converting the K-class clustering into a sequence of K−1 much simpler binary clustering. • Independent to initialization • Very close good local minima obtained by α-expansion and α-β swap

30. Segmentation Evolution Iter #1 Iter #2 Iter #3 Iter #4 Mean image

31. Outline • Motivation • Graph formulation of MRF labeling • Graduated graph cuts • Experimental results • Conclusion

32. Comparative Results Comparative Experiments

33. Robustness to Noise Robust to noise

34. Preservation of Soft Boundary

35. Consistency to Ground Truth

36. Coarse-to-Fine Segmentation

37. Performance Summary

38. Outline • Motivation • Graph formulation of MRF labeling • Graduated graph cuts • Experimental results • Conclusion

39. Conclusion • An efficient self-validated labeling method that is very close to good local minima and guarantees stepwise global optimum • Provides a vertical extension to binary graph cut that is independent to initialization • Ready to apply to a wide range of clustering problems in low-level vision

40. Thanks!