440 likes | 584 Views
The Layout Consistent Random Field for detecting and segmenting occluded objects. John Winn Jamie Shotton. CVPR, June 2006. LayoutCRF contributions. Detection and segmentation Handles occlusion and deformation Multiple objects simultaneously Multiple classes. Roadmap. Related work
E N D
The Layout Consistent Random Fieldfor detecting and segmenting occluded objects John Winn Jamie Shotton CVPR, June 2006
LayoutCRF contributions • Detection and segmentation • Handles occlusion and deformation • Multiple objects simultaneously • Multiple classes
Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results
Related work: constellation models X [Fergus et al. CVPR 2003] [Leibe et al. ECCV 2004] [Crandall et al. ECCV 2006] [Kumar et al. CVPR 2005] …
Related work: constellation models X X X X [Fergus et al. CVPR 2003] [Leibe et al. ECCV 2004] [Crandall et al. ECCV 2006] [Kumar et al. CVPR 2005] …
Related work: windowed detectors Classifier Car Localised features Sliding window [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…
Related work: windowed detectors Classifier Car? Wall? Localised features Sliding window [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…
Related work: multiclass segmentation TextonBoost [Shotton et al. ECCV 2006] building tree car road Doesn’t exploit layout of parts – can’t identify object instances [Tu et al. CVPR 2003] [He et al. CVPR 2004]
Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results
Dense part labelling Automatic per-pixel labelling based on a grid of parts Part labels (color-coded)
Dense part labelling Background label Part labels (color-coded)
Patch-based part detector • Decision forest classifier • Features are differences of pixel intensities Classifier [Lepetit et al. CVPR 2005]
Decision trees Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities. Improved performance with multiple decision trees (random forest). Performs as well as boosting with shared features, but can process much more data in the same time.
Colors show posterior over part labels – part detectors are noisy! Part color key Patch-based part detector
Layout consistency Neighboring pixels (7,2) (8,2) (9,2) (p,q) ? (7,3) (8,3) (9,3) (7,4) (8,4) (9,4)
Layout consistency (7,2) (8,2) (9,2) (7,3) (8,3) (9,3) Layoutconsistent (7,4) (8,4) (9,4) Neighboring pixels (p,q) (p,q) (p+1,q) (p+1,q-1) Allows for deformation/rotation (p+1,q+1)
Layout consistency (7,2) (8,2) (9,2) (7,3) (8,3) (9,3) (7,4) (8,4) (9,4) Neighboring pixels (p,q) ? (p,q) (p-1,q+1) (p,q+1) (p+1,q+1) Layoutconsistent
Object occludes background (object edge) ‘Background’ occludes object One object instance occludes another Occlusions Not layout consistent = occlusion (or invalid)
Effect of layout consistency Input image Part detector output Layout consistent regions With layout consistency
Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results
Layout Consistent Random Field Part detector Image I Part labels h
Layout consistency Part detector Image I Layout Consistent Random Field Part labels h
Layout Consistent Random Field Layout consistency Part detector Edge weight Parameters θ’={ βbg , βoe , βco , βiif , e0 , γ} (set by hand)
Inference of MAP labelling Proposed labelling Graph cuts with customised alpha-expansion move Part labels h [Boykov and Jolly, ICCV 2001]
Inference of MAP labelling Graph cuts with customised alpha-expansion move Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]
Inference of MAP labelling Graph cuts with customised alpha-expansion move Expansion move not accepted Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]
Inference of MAP labelling Graph cuts with customised alpha-expansion move Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]
Decision tree re-learning Part-labels are inferred (constrained by known mask) and decision forest re-trained
Limitation of layout consistency • Allows arbitrary stretching/scaling
Global layout Global layout InstanceT2 Image I Part labels h Global layout constraint is (weak) star-shaped constellation model Constrains part locationsrelative to centroid Instance T1 Allows competition between different object instances
T3 T1 T1 T2 T2 Example with global consistency Input image Layout consistent regions Instance labelling
Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results
UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)
UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)
UIUC car database: detection Results refer to detection of unoccluded cars only.
Detecting heavily occluded faces • Caltech face database with artificial occlusions • AR face database with real occlusions
Stability of part labelling Part color key
Multi-class detection • Can extend to multiple classes with different numbers of part labels for each class • Example: building has multiple parts, other classes have one
Summary + future directions Summary: LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects Future directions: • Extend to multiple viewpoints and multiple scales • Share parts between classes • Incorporate object context (‘car above road’) • Incorporate geometric cues
Thank you jwinn@microsoft.com http://johnwinn.org/