1 / 36

Associative Hierarchical CRFs for Object Class Image Segmentation

Associative Hierarchical CRFs for Object Class Image Segmentation. International Conference on Computer Vision (ICCV) 2009. L’ubor Ladick’y and Chris Russell Oxford Brookes University. Pushmeet Kohli Microsoft Research Cambridge. Philip H.S. Torr Oxford Brookes University. Outline.

carol
Download Presentation

Associative Hierarchical CRFs for Object Class Image Segmentation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Associative Hierarchical CRFs for Object Class Image Segmentation International Conference on Computer Vision (ICCV)2009 L’uborLadick’yand Chris Russell Oxford Brookes University PushmeetKohli Microsoft Research Cambridge Philip H.S. Torr Oxford Brookes University

  2. Outline • Introduction • Random Fields for Labelling Problems • Hierarchical CRF for Object Segmentation • Experiments and Results

  3. Structure

  4. Introduction

  5. Region of interest Mean Shift Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls

  6. Region of interest Mean Shift Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls

  7. Region of interest Mean Shift Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls

  8. Region of interest Mean Shift Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls

  9. Region of interest Mean Shift Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls

  10. Region of interest Mean Shift Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls

  11. Region of interest Mean Shift Center of mass Objective : Find the densest region Distribution of identical billiard balls

  12. Pixel V.S. Segment • The difference of pair CRFs between based on pixels and segments • Based on pixels: • No quantization errors • Lack of long range interactions • Results oversmoothed • Based on segments • Allows long range interactions • Can not recover from incorrect segmentation

  13. Introduction • Propose a novel hierarchical CRF • Integration of features derived for different quantisation levels • Propose new sophisticated potentials defined over the different levels of the quantisation hierarchy • Use a novel formulation that allows context to be incorporate at multiple levels of multiple quantisation

  14. Random Fields for Labelling Problems • Introduce the pixel-based CRF used for formulation the object class segmentation problem • One discrete R.V. per image pixel, each of which may take a value from the set of labels • Symbols: • Label: • R.V. : • Pixel • The set of all neighbours of the variable : • A clique c is a set of random variables • Labelling: denoted by take the value from

  15. CRFs D: the set of the data, Z: the partition function, C: the set of all cliques : the potential function of the clique , • The energy form: • The most probable or MAP labeling : • Wrote as the sum of unary and pairwise potentials

  16. The Robust Model • Extended by the robust potentials [KohLi et al., 2008 ] S:the set of the segments • The pixels within the same segment are more like likely to take the same label • the form of the robust potentials: ; • The weighted version:

  17. -Based Hierarchical CRFs • The single auxiliary variable where c is a segment or a clique • Take the value from an extended label set

  18. -Based Hierarchical CRFs • New cost function over • The unary potential over Y, • The pairwise potential • over Y and X • Goal: • The new energy function:

  19. Recursive Form • The auxiliary variables in the last layer are the input variable • The new energy function: • The recursive form: • Initial form:

  20. Hierarchical CRF for Object Segmentation • Describe the set of potentials in the object-class segmentation problem • Include unary potentials for both pixels and segments, pairwise potentials between pixels and segments and connective potentials between pixels and their containing segments

  21. Robustness to misleading segmentations • Using unsupervised segmentation algorithm may be misleading – segment may contain multiple object classed • Assigning the same label to all pixels will result in an incorrect labeling • Overcome it by using the segment quality measures [Rabinovichet at., 2009] and [Ren and Malik, 2003] • By modifying the potentials according to a quality sensitive measure for all segment c • Writing weight features based potential over c

  22. Potentials for object class segmentation • Refer to elements of each layer as pixels, segments, and super-segments • At the pixel level: • The unary potentials are computed using a boosted dense feature classifier [Shotton et al., 2006] • The pairwise potentials [Boykov and Jolly, 2001] , [Rother at al., 2004]:

  23. Potentials for object class segmentation • At the segment level: • Initially found using a fine scale mean-shift algorithm [Comaniciu and Meer, 2002] • Contain little novel local information, but strong predictors of consistency • The potentials learning at this level are uniform, due to the lack of unique features, however as they are strongly indicative of local consistency, the penalty associated with breaking them is high • To encourage neighbouring segments with similar texture to take the same label, used pairwise potentials based on the Euclidean distance if normalized histograms of colour

  24. Potentials for object class segmentation • At the super-segment level: • Based upon a coarse mean-shift segmentation, performed over the result of the previous segmentations • Contain significantly more internal information than their smaller children • Propose unary segment potential based on the histograms of features

  25. Unary Potentials From Dense Feature • Perform texture based segmentation at pixel level • Derived from TextonBoost[Shotton et al., 2006] • The features are computed on every pixel • Extend the TextonBoostby boosting classifiers defined on multiple dense feature together • Dense–feature shape filters defined by triplets: [f, t, r] where f is a feature type, t is a feature cluster, and r is a rectangular region • Feature response : Given a point i, the number of features of type f belongto the cluster t in the region r relative to the point i

  26. Histogram-Based Segment Unary Potentials • Defined over segments and super-segments • The distribution of dense features responses are more discriminative than any feature along • The unary potential of an auxiliary variable representing a segment is learnt by (using the normalized histograms of multiple clustered dense features) using multi-class Gentle Ada-boost[Torralba et al., 2004] • Weak classifiers: f:the normalized histogram of the feature set t: the cluster index a: threshold

  27. Histogram-Based Segment Unary Potentials • The segment potential: a :the response given by the Ada-boost classifier to clique c taking label l a : the truncation threshold a , and a normalizing constant

  28. Learning Weights for Hierarchical CRFs • Uses a coarse to fine, layer-based, local search scheme over a validation set • Introduce additional notation: • :the variable contained in the layer • : the labelling of associated with a MAP estimate • Determine a dominant label for each segment c, such that when , if there is no such dominant label, set a • The label of a clique : correspond to the dominant label of this clique(segment) in the ground truth (or ) for its containing ot be correctly labelled

  29. Learning Weights for Hierarchical CRFs • At eachlayer, seek to minimize the discrepancy between the dominant ground truth of a clique(segment), and the value of the MAP estimate • Choose parameters λ to minimize

  30. Algorithm : the weighting of unary terms in the layer a : the weighting of pairwise terms in the layer a : a scalar modifier of all terms in the layer a : an arbitrary constant that controls the precision of the final assignment of

  31. Experiments • Two data sets • MSRC-21 [Shottonet al., 2006] • Resolution: pixels • 21 object classes • PASCAL VOC 2008 [Everingham et al., 2008, website] • 511 training, 512 validation and 512 segmented test images • 20 foreground and 1 background classes • 10, 057 images for which only the bounding boxes of the objects present in the image are marked

  32. Results on The MSRC-21

  33. Results on The MSRC-21 [25]: J. Shottonet al., CVPR, 2008 [26]: J. Shottonet al., ECCV, 2006 [1]: D. Batraet al., CVPR, 2008 [25]: L. Yang et al., CVPR, 2007

  34. Results on The MSRC-21 [25]: J. Shottonet al., CVPR, 2008 [26]: J. Shottonet al., ECCV, 2006 [1]: D. Batraet al., CVPR, 2008 [25]: L. Yang et al., CVPR, 2007

  35. Results on The VOC-2008

  36. Results on The VOC-2008

More Related