1 / 46

What, Where & How Many? Combining Object Detectors and CRFs

What, Where & How Many? Combining Object Detectors and CRFs. Ľ ubor Ladick ý , Paul Sturgess, Karteek Alahari, Christopher Russell, Philip H.S. Torr. http://cms.brookes.ac.uk/research/visiongroup/. Complete Scene Understanding. Involves

pennie
Download Presentation

What, Where & How Many? Combining Object Detectors and CRFs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. What, Where & How Many?Combining Object Detectors and CRFs Ľubor Ladický, Paul Sturgess, Karteek Alahari, Christopher Russell, Philip H.S. Torr http://cms.brookes.ac.uk/research/visiongroup/

  2. Complete Scene Understanding Involves • Localization of all instances of foreground objects (“things”) • Localization of all background classes (“stuff”) • Pixel-wise segmentation • 3D reconstruction • Pose detection • Action recognition • Event recognition

  3. Complete Scene Understanding Involves • Localization of all instances of foreground objects (“things”) • Localization of all background classes (“stuff”) • Pixel-wise segmentation • 3D reconstruction • Pose detection • Action recognition • Event recognition Semantic Segmentation

  4. Complete Scene Understanding Involves • Localization of all instances of foreground objects (“things”) • Localization of all background classes (“stuff”) • Pixel-wise segmentation

  5. Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. Adelson, Forsyth et al. 96

  6. Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. Adelson, Forsyth et al. 96

  7. Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. Adelson, Forsyth et al. 96

  8. Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. Adelson, Forsyth et al. 96

  9. Semantic Scene Understanding We're interested in whole scene understanding Given an image, detect every thing in it. Thing : An object with a specific size and shape. Adelson, Forsyth et al. 96

  10. Semantic Scene Understanding We're interested in whole scene understanding Given an image, label all the stuff Stuff : Material defined by a homogeneous or repetitive pattern, with no specific spatial extent / shape. Adelson, Forsyth et al. 96

  11. Semantic Scene Understanding We're interested in whole scene understanding Given an image, label all the stuff Stuff : Material defined by a homogeneous or repetitive pattern, with no specific spatial extent / shape. Adelson, Forsyth et al. 96

  12. Semantic Scene Understanding Our plan is to combine • State of the art sliding window object detection • State of the art segmentation techniques car

  13. Algorithms for Object Localization Sliding window detectors • HOG descriptor (Dalal & Triggs CVPR05) • Based on histograms of features (Vedaldi et al. ICCV09) • Part-based models (Felzenszwalb et al. CVPR09)

  14. Algorithms for Object Localization Sliding window detectors • HOG descriptor (Dalal & Triggs CVPR05) • Based on histograms of features (Vedaldi et al. ICCV09) • Part-based models (Felzenszwalb et al. CVPR09)

  15. Algorithms for Object Localization Sliding window detectors • HOG descriptor (Dalal & Triggs CVPR05) • Based on histograms of features (Vedaldi et al. ICCV09) • Part-based models (Felzenszwalb et al. CVPR09)

  16. Algorithms for Object Localization Non-maxima suppression • Greedily • Using field of indicator variables (Ramanan et al ICCV09) • One binary variable yi{ 0, 1 } per location/scale

  17. Algorithms for Object Localization car Non-maxima suppression • Greedily • Using field of indicator variables (Ramanan et al ICCV09) • One binary variable yi{ 0, 1 } per location/scale

  18. Sliding window detectors car • Sliding window + Segmentation • OBJCUT (Kumar et al. 05) • Updating colour model (GrabCut - Rother et al. 04)

  19. Sliding window detectors Sliding window detectors not good for “stuff” Try to detect “sky” !

  20. Sliding window detectors Sliding window detectors not good for “stuff” sky Sky is irregular shape not suited to the sliding window approach

  21. Sliding window detectors • State-of-the-art for object detection (“things”) • Do not work for background classes (“stuff”) • No distinct shape • Cannot be enclosed in a box • Cannot recover from incorrect detections

  22. Algorithms for Object-class Segmentation Pairwise CRF over pixels CRF construction Input image Final segmentation Shotton et al. ECCV06

  23. Algorithms for Object-class Segmentation Pairwise CRF over pixels CRF construction Input image Training of Potentials Shotton et al. ECCV06

  24. Algorithms for Object-class Segmentation Pairwise CRF over pixels CRF construction Input image Training of Potentials MAP Final segmentation Shotton et al. ECCV06

  25. Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenszwalb, Huttenlocher, IJCV2004

  26. Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenszwalb, Huttenlocher, IJCV2004

  27. Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image Training of potentials Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10

  28. Algorithms for Object-class Segmentation Pairwise CRF over Super-pixels / Segments Unsupervised segmentation Input image Training of potentials MAP Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10 Final segmentation

  29. Algorithms for Object-class Segmentation Associative Hierarchical CRF Multiple segmentations or hierarchies Input image Ladický et al. ICCV09

  30. Algorithms for Object-class Segmentation Associative Hierarchical CRF Multiple segmentations or hierarchies CRF construction Input image Ladický et al. ICCV09

  31. Algorithms for Object-class Segmentation Associative Hierarchical CRF Multiple segmentations or hierarchies CRF construction Input image MAP Final segmentation Ladický et al. ICCV09, Russell et al. UAI10

  32. Associative Hierarchical CRF • State-of-the-art performance on MSRC & CamVid • Merges information at multiple scales • Can recover from incorrect segmentations Ladický et al. ICCV09, Sturgess et al., BMVC09

  33. Associative Hierarchical CRF • State-of-the-art performance on MSRC & CamVid • Merges information at multiple scales • Can recover from incorrect segmentations However, • No concept of “things” • Cannot distinguish between multiple instances car Ladický et al. ICCV09, Sturgess et al., BMVC09

  34. CRF Formulation with Detectors • CRF formulation altered with a potential for each detection

  35. CRF Formulation with Detectors • CRF formulation altered with a potential for each detection AH-CRF energy without detectors CRF graph over pixels

  36. CRF Formulation with Detectors • CRF formulation altered with a potential for each detection AH-CRF energy without detectors Set of pixels of d-th detection Classifier response Detected label CRF graph over pixels

  37. CRF Formulation with Detectors • Joint CRF formulation should contain • Possibility to reject detection hypothesis • Recover the status of the detection (0 / 1) • Thus, potential is a minimum over indicator variable yd { 0, 1 } Indicator variables CRF graph over pixels

  38. CRF Formulation with Detectors • Detection potential should decrease the energy of labelling agreeing with the detection hypothesis • Partial disagreement penalized but not directly rejects the hypothesis Indicator variables CRF graph over pixels

  39. CRF Formulation with Detectors • Detection potential should decrease the energy of labelling agreeing with the detection hypothesis • Partial disagreement penalized but not directly rejects the hypothesis Detection hypothesis status Partial inconsistency cost Detection strength CRF graph over pixels

  40. CRF Formulation with Detectors • Detection potential should decrease the energy of labelling agreeing with the detection hypothesis • Partial disagreement penalized but not directly rejects the hypothesis Linear Linear thresholded CRF graph over pixels

  41. Inference over CRF with Detectors • This higher order potential can be transformed to • Solvable with all graph cut-based methods which take the form of Robust PN (Kohli et al. CVPR08)

  42. Results on CamVid dataset Result without detections Set of detections Final Result Brostow et al. Sturgess et al. Brostow et al. ECCV08, Sturgess et al. BMVC09

  43. Results on CamVid dataset Result without detections Set of detections Final Result Also provides number of object instances (using yd’s)

  44. Results on VOC2009 dataset CRF without detectors CRF with detectors CRF without detectors CRF with detectors Input image Input image

  45. Summary • Novel formulation of CRF with detectors • Can recover from incorrect detections • Possible to obtain instances of objects • Efficiently solvable

  46. Future and ongoing work • Automatic non-maximum suppression • Part-based models in CRF • New things & stuff dataset available soon! • Questions?

More Related