Figure/Ground Assignment in Natural Images

1. Figure/Ground AssignmentinNatural Images Xiaofeng Ren, Charless Fowlkes and Jitendra Malik University of California, Berkeley Computer Vision Group ECCV 2006 Graz

2. Figure/Ground Organization

3. Ground (shapeless) Figure (face) Figure (Goblet) Ground (Shapeless) Figure/Ground Organization • A contour belongs to one of the two (but not both) abutting regions. Important for the perception of shape

4. A Classical View of Perception Object and Scene Recognition Figure/Ground Organization Grouping / Segmentation Image

5. A Modern View Object and Scene Recognition Grouping / Segmentation Figure/Ground Organization

6. Focus of This Work Object and Scene Recognition Grouping / Segmentation Figure/Ground Organization

7. Overview • Introduction • Groundtruth for Figure/Ground • Local Figure/Ground Cues • Shapemes • Global Consistency • Conditional Random Field • Results and Quantitative Evaluation • Conclusion

8. Figure/Ground in Natural Images

9. G F Figure/Ground: Groundtruth

10. Figure/Ground: Groundtruth

11. Figure/Ground Dataset [Martin, Fowlkes & Malik; ECVP 2003]

12. Cues for Figure/Ground • Local Cues • Gestalt Principles of Figure/Ground • Global Cues • Label Consistency at T-junctions [Kienker, Sejnowski, Hinton & Schumacher 1986] [Heitger & von der Heydt 1993] [Geiger, Kumaran & Parida 1996] [Saund 1999] [Yu, Lee and Kanade 2001] … …

13. local shapes …… Shapemes: Prototypical Local Shapes collect Use Geometric Blur [Berg & Malik 2001] cluster

14. parallelism straight line convexity line ending corner

15. G F G F G Gestalt Principles for Figure/Ground • Convexity • Parallelism • Surroundedness • Symmetry • Lower Region • Common Fate ……

16. L R L:93.8% L:89.6% L:66.5% L:49.8% L:11.7% L: 5.0% Shapemes for F/G Discrimination Which side is Figure? Train a logistic classifier to linearly combine the shapeme cues

17. F F G G G F G G F G F F common uncommon Global Consistency

18. Edge potentialsexp(ii) Junction potentialsexp(jj) where Conditional Random Fields (CRF) X={X1,X2,…,Xm} Xi{Left,Right}

19. Building a CRF Model • What are the features? • edge features: • Shapemes • junction features: • Junction type • How to make inference? • Loopy Belief Propagation • How to learn the parameters? • Gradient Descent on Max. Likelihood X={X1,X2,…,Xm} Estimate P(Xi|)

20. F G G G G G F F F F { (G,F),(F,G) } F G F G G F { (F,G),(F,G),(F,G) } Junction Features • One feature for each junction type { (F,G),(G,F),(F,G) } Junction potentials:

21. G F G G G F F F = -0.611 = 0.185 F F G G F G G G F G F F = 0.428 = -0.857 Learning Junction Weights

22. CSP > : contour direction + : convex edge - : concave edge possible junctions (constraints) Line Labeling • Probabilistic constraints instead of logical constraints • Training and testing on large datasets of natural images [Clowes 1971, Huffman 1971; Waltz 1972; Malik 1986]

23. Experiments • Using human-marked segmentations • Using edges computed by an edge detector

24. 50% 50% 88% 88% Results 55.6% ------ 64.8% 64.9% 72.0% 66.5% 78.3% 68.9%

25. Using human segmentations Image Groundtruth Local Global

26. Using edge maps computed from an edge detector Image Edge Map Local Global

27. Summary • Human subjects label groundtruth figure/ground assignments in natural images. • Shapemes encode Gestalt structures and capture local figure/ground cues. • A conditional random field enforces global consistency at T-junctions and improves figure/ground performance. • Quantitative evaluations show that figure/ground organization has a promising solution. Object and Scene Recognition Grouping / Segmentation Figure/Ground Organization

28. Future Work Object and Scene Recognition Grouping / Segmentation Figure/Ground Organization

29. Future Work Object and Scene Recognition Grouping / Segmentation Figure/Ground Organization

30. Thank You

34. Baseline: Size/Convexity Size(A) < Size(B): A is figure; B is ground Size(A) > Size(B): B is figure; A is ground B A

35. F F G G G G F F G F F G   Continuity in Figure/Ground If a pair of edges belong to the same foreground, they should have a smooth connection.