1 / 30

Real-time Foreground Extraction with RGBD Camera for 3D Telepresence

Real-time Foreground Extraction with RGBD Camera for 3D Telepresence. Presenter: ZHAO Mengyao, PhD, SCE Supervisor: Asst/P FU Chi-Wing, Philip Co-Supervisor: A/P CAI Jianfei. Outline. Motivation Related Work Challenges Our Approach Results Limitation

sydney
Download Presentation

Real-time Foreground Extraction with RGBD Camera for 3D Telepresence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Real-time Foreground Extraction with RGBD Camera for 3D Telepresence Presenter: ZHAO Mengyao, PhD, SCE Supervisor: Asst/P FU Chi-Wing, Philip Co-Supervisor: A/P CAI Jianfei

  2. Outline • Motivation • Related Work • Challenges • Our Approach • Results • Limitation • Future Work

  3. 3D Telepresence

  4. 3D Telepresence BeingThere Centre’s RBT Project IBM’s Holographic 3D cell phone

  5. Foreground Extraction

  6. Related Work • Chroma Key [1, 2]

  7. Related Work • Interactive Approach [3-8]

  8. Related Work Microsoft Kinect [9] PrimeSense Carmine 3D Sensor [10]

  9. Related Work • Real-time Foreground Extraction with RGBD camera: • FreeCam [11]

  10. Challenges • Convenience • Arbitrary background • High quality • Natural & smooth boundary • Automation • No manual markup • Real-time • Support teleconference/telepresence • Temporal coherency • Free of flickering artifact

  11. Challenges • Inaccurate depth map Red: Color Green: Depth • Noisy depth map • Depth/Color not well aligned • Depth/Color not synchronized Depth on Color

  12. Our Approach • We aim to • Perform high-quality coherent foreground extraction in real-time that could support teleconference and telepresence • We propose • An integrated pipeline for robust foreground extraction with RGBD camera • A temporal coherent matting approach • A CUDA based GPU implementation of our approach that achieves real-time performance

  13. Our Approach – Matting where iis the index of pixel, I is the intensity, αis the alpha, F is the foreground, B is the background Input Trimap Alpha

  14. Our Approach – Workflow

  15. Our Approach – Pipeline

  16. Our Approach – Pipeline

  17. Our Approach – Temporal Hole Filling with Depth Map Shadow Detection NMD: no-measured depth [12] Black: NMD regions Green: out-of-range regions Yellow: mirror-like regions Red: shadow regions Raw depth map Detected shadow Types of NMD regions

  18. Our Approach – Temporal Hole Filling with Depth Map Shadow Detection • Temporal Hole-filling: • For shadow region, apply below: • For other NMD region, apply joint-bilateral filter

  19. Our Approach – Pipeline

  20. Our Approach –Adaptive Binary Mask Generation color mask final mask depth mask

  21. Our Approach – Pipeline

  22. Our Approach –Non-local Temporal Matting: intro of closed-form matting • Closed-form matting: • Assumption: • Both F and B are approximately constant over a small window around each pixel. 1 2 • Alpha can be obtained by solving: ( 3 )

  23. Our Approach –Non-local Temporal Matting: extension of closed-form matting Closed-form matting Assumption: F and B are smooth in local window Temporal coherency Assumption: F and B are smooth in both spatial and temporal domain Non-local temporal matting

  24. Our Approach –Non-local Temporal Matting: 3d non-local neighbor It+1 It0 It-1 It0 2D neighbor 3D non-local neighbor is the neighbor of (4) (5) (3)

  25. Our Approach – Non-local Temporal Matting: volume partition • The linear equation system is too large • Use kd-tree segmentation to partition the volumes • Recursive until number of unknown within each block is smaller than a threshold

  26. Our Approach –GPU implementation: CUDA and CULA Sparse Quantitative Performance

  27. ResultsComparison with three other state-of-the-art works

  28. Limitation • Shadow-like region when moving fast • Reason 1: color/depth not well aligned • Reason 2: color/depth not synchronized • Sometimes inaccurate when foreground/background share similar color

  29. Future Work • Refine the depth map using attained alpha map to achieve better 3D representation

  30. Thank You! Q & A

More Related