1 / 33

3D-Based Reasoning with Blocks, Support, and Stability

3D-Based Reasoning with Blocks, Support, and Stability. Zhaoyin Jia. School of Electrical and Computer Engineering Cornell University. Computer Vision with RGB-D. Pose Recognition J. Shotton et al. 2011; G. Girshick et al. 2013. Activity Detection

gotzon
Download Presentation

3D-Based Reasoning with Blocks, Support, and Stability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3D-Based Reasoning with Blocks, Support, and Stability Zhaoyin Jia School of Electrical and Computer Engineering Cornell University

  2. Computer Vision with RGB-D Pose Recognition J. Shotton et al. 2011; G. Girshick et al. 2013. Activity Detection J. Sung et al. 2012; H. Koppula et al. 2013. Object Recognition K. Lai et al. 2011; A. Janoch et al. 2011 3D Scene Labeling H. Koppula, et al. 2011; N. Silberman et al 2011, 2012. Jia, Gallagher, Saxena and Chen

  3. RGB-D Images Jia, Gallagher, Saxena and Chen

  4. 3D Reasoning on RGB-D Images • Free Space: • objects can be placed in empty spaces. • Physical Stability: • one book is supported by the table and wall. • Foresee Consequences: • the camera and the book will fall if the box moves. Jia, Gallagher, Saxena and Chen

  5. Reasoning with Blocks, Support, & Stability Input: RGB-D Segmentation Jia, Gallagher, Saxena and Chen

  6. Reasoning with Blocks, Support, & Stability Input: RGB-D Blocks, Support, and Stability Jia, Gallagher, Saxena and Chen

  7. Reasoning with Blocks, Support, & Stability Input: RGB-D Final 3D representation Jia, Gallagher, Saxena and Chen

  8. Algorithms Jia, Gallagher, Saxena and Chen

  9. Overview 3D Block Fitting Support and Stability Input Segmentation* Evaluate Energy Function * "Indoor Segmentation and Support Inference from RGBD Images," N. Silberman et al. ECCV, 2012. Jia, Gallagher, Saxena and Chen

  10. Overview 3D Block Fitting Input Segmentation 3D Block Fitting Support and Stability Evaluate Energy Function Jia, Gallagher, Saxena and Chen

  11. Single Block Fitting • 3D orientated bounding box on depth data • Partially observed. Minimum volume may fail * • Minimum surface distance (Min-surf) * "Fast oriented bounding box optimization on the rotation group SO(3, R)," C. Chang et al, ACM Transactions on Graphics, 2011. Jia, Gallagher, Saxena and Chen

  12. Overview 3D Block Fitting Input Segmentation Support and Stability Support and Stability Evaluate Energy Function Jia, Gallagher, Saxena and Chen

  13. Support and Stability Support Relations Supporting Area Stability

  14. Support Relation Surface On-top Support Partial On-top Support Side Support Jia, Gallagher, Saxena and Chen

  15. Separate axis is perpendicular to y Separate axis is parallel to y Surface On-top Support Partial On-top Support Side Support Jia, Gallagher, Saxena and Chen

  16. Surface On-top Support Partial On-top Support Side Support Jia, Gallagher, Saxena and Chen

  17. From Support To Stability • Supporting Area Jia, Gallagher, Saxena and Chen

  18. From Support To Stability • Supporting Area • Stability Stable Jia, Gallagher, Saxena and Chen

  19. From Support To Stability • Supporting Area • Stability Stable Unstable Jia, Gallagher, Saxena and Chen

  20. Overview 3D Block Fitting Input Segmentation Support and Stability Evaluate Energy Function Evaluate Energy Function Jia, Gallagher, Saxena and Chen

  21. Reasoning Through an Energy Function Segmentation Energy Function Use Support Relations, Stability, Other Box-based/RGB-D info as features. Better Segmentation Smaller F(S) Worse Segmentation Larger F(S) RGB-D Jia, Gallagher, Saxena and Chen

  22. Energy Function: Single Box Potential • Features: minimum surface distance, visibility, single box stability, etc. Worse Box Better Box Jia, Gallagher, Saxena and Chen

  23. Energy Function: Pairwise Box Potential • Features: box intersection, support, supporting area distance etc. Worse Boundary Better Boundary Jia, Gallagher, Saxena and Chen

  24. 1.4 …… …… 2.3 Segmentation Energy Function: Segmentation at one step …… 1.2 …… …… Jia, Gallagher, Saxena and Chen

  25. Summary 3D Block Fitting Support and Stability Input Segmentation Evaluate Energy Function Jia, Gallagher, Saxena and Chen

  26. Experiments Jia, Gallagher, Saxena and Chen

  27. Experiments: • Block dataset • Cornell Support Object dataset (SOD) • 300 RGB-D images with ground-truth segments and support relations • NYU-2 RGB-D dataset Jia, Gallagher, Saxena and Chen

  28. Experiment: Segmentation Results • Pixel-wise object segmentation accuracy: Jia, Gallagher, Saxena and Chen

  29. Experiment: Segmentation Results Input RGB-D images Jia, Gallagher, Saxena and Chen

  30. Experiments: Support Inference • Neighbor: object is supported by its neighbors • Stability: trim unnecessary support after reasoning Jia, Gallagher, Saxena and Chen

  31. Color Segmentation D. Hoiem et al. ICCV, 2007; P. Arbelaez et al. CVPR, 2012. …… Semantic 3D Labeling H. Koppula et. al. NIPS 2011. Blocks world revisited A. Gupta et all, ECCV, 2010. Object Placement Y. Jiang et al. IJRR, 2012. Indoor Segmentation & Support N. Silberman et al. ECCV 2012. Jia, Gallagher, Saxena and Chen

  32. Conclusion • 3D support and stability • Based on box representations • Object segmentation in 3D scene • Learning algorithm. • Future work • Non-uniform density • Semantic classification on blocks • Occluded supports Jia, Gallagher, Saxena and Chen

  33. 3D-Based Reasoning with Blocks, Support, and Stability Zhaoyin Jia, Andrew Gallagher, AshutoshSaxena, Tsuhan Chen Cornell University Thanks. Questions?

More Related