The Layout Consistent Random Field for detecting and segmenting occluded objects

1. The Layout Consistent Random Fieldfor detecting and segmenting occluded objects John Winn Jamie Shotton CVPR, June 2006

2. LayoutCRF contributions • Detection and segmentation • Handles occlusion and deformation • Multiple objects simultaneously • Multiple classes

3. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

4. Related work: constellation models X [Fergus et al. CVPR 2003] [Leibe et al. ECCV 2004] [Crandall et al. ECCV 2006] [Kumar et al. CVPR 2005] …

5. Related work: constellation models X X X X [Fergus et al. CVPR 2003] [Leibe et al. ECCV 2004] [Crandall et al. ECCV 2006] [Kumar et al. CVPR 2005] …

6. Related work: windowed detectors Classifier Car Localised features Sliding window [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…

7. Related work: windowed detectors Classifier Car? Wall? Localised features Sliding window [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…

8. Related work: multiclass segmentation TextonBoost [Shotton et al. ECCV 2006] building tree car road Doesn’t exploit layout of parts – can’t identify object instances [Tu et al. CVPR 2003] [He et al. CVPR 2004]

9. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

10. Dense part labelling Automatic per-pixel labelling based on a grid of parts Part labels (color-coded)

11. Dense part labelling Background label Part labels (color-coded)

12. Patch-based part detector • Decision forest classifier • Features are differences of pixel intensities Classifier [Lepetit et al. CVPR 2005]

13. Decision trees Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities. Improved performance with multiple decision trees (random forest). Performs as well as boosting with shared features, but can process much more data in the same time.

14. Colors show posterior over part labels – part detectors are noisy! Part color key Patch-based part detector

15. Layout consistency

16. Layout consistency

17. Layout consistency Neighboring pixels (7,2) (8,2) (9,2) (p,q) ? (7,3) (8,3) (9,3) (7,4) (8,4) (9,4)

18. Layout consistency (7,2) (8,2) (9,2) (7,3) (8,3) (9,3) Layoutconsistent (7,4) (8,4) (9,4) Neighboring pixels (p,q) (p,q) (p+1,q) (p+1,q-1) Allows for deformation/rotation (p+1,q+1)

19. Layout consistency (7,2) (8,2) (9,2) (7,3) (8,3) (9,3) (7,4) (8,4) (9,4) Neighboring pixels (p,q) ? (p,q) (p-1,q+1) (p,q+1) (p+1,q+1) Layoutconsistent

20. Object occludes background (object edge) ‘Background’ occludes object One object instance occludes another Occlusions Not layout consistent = occlusion (or invalid)

21. Effect of layout consistency Input image Part detector output Layout consistent regions With layout consistency

22. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

23. Layout Consistent Random Field Part detector Image I Part labels h

24. Layout consistency Part detector Image I Layout Consistent Random Field Part labels h

25. Layout Consistent Random Field Layout consistency Part detector Edge weight Parameters θ’={ βbg , βoe , βco , βiif , e0 , γ} (set by hand)

26. Inference of MAP labelling Proposed labelling Graph cuts with customised alpha-expansion move Part labels h [Boykov and Jolly, ICCV 2001]

27. Inference of MAP labelling Graph cuts with customised alpha-expansion move Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]

28. Inference of MAP labelling Graph cuts with customised alpha-expansion move Expansion move not accepted Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]

29. Inference of MAP labelling Graph cuts with customised alpha-expansion move Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]

30. Example inference

31. Decision tree re-learning Part-labels are inferred (constrained by known mask) and decision forest re-trained

32. Limitation of layout consistency • Allows arbitrary stretching/scaling

33. Global layout Global layout InstanceT2 Image I Part labels h Global layout constraint is (weak) star-shaped constellation model Constrains part locationsrelative to centroid Instance T1 Allows competition between different object instances

34. T3 T1 T1 T2 T2 Example with global consistency Input image Layout consistent regions Instance labelling

35. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

36. UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

37. UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

38. UIUC car database: detection Results refer to detection of unoccluded cars only.

39. Detecting heavily occluded faces • Caltech face database with artificial occlusions • AR face database with real occlusions

40. Stability of part labelling Part color key

41. Multi-class detection • Can extend to multiple classes with different numbers of part labels for each class • Example: building has multiple parts, other classes have one

42. Summary + future directions Summary: LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects Future directions: • Extend to multiple viewpoints and multiple scales • Share parts between classes • Incorporate object context (‘car above road’) • Incorporate geometric cues

43. Thank you jwinn@microsoft.com http://johnwinn.org/