High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image

1. ICPR/WDIA-2012 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung Lie National Chung Cheng University Taiwan

2. Outline • Introduction • Our hybrid camera system • The proposed depth processing algorithm • Experimental results • Conclusion

3. Color + Depth in 3DTV/FTV • Depth image, in combination with DIBR (Depth Image-Based Rendering) technique, is important in multi-view 3DTV/FTV applications. Depth image Colorimage DIBR view9 view8 view7 view6 view5 view4 view3 view2 view1

4. Depth acquisition • From stereo matching via disparity estimation • Unreliable for textureless regions • Computationally intensive • From active photo-electrical sensing • Lower resolution • Higher acquisition speed and accuracy Kinect SR4000 ToF

5. Hybrid camera systems • High-resolution color cameras + Low-resolution depth camera (mis-alignment between their optical axis) • [12]: Use warped data of depth camera to refine ROI disparity of the dual-eye camera • Proposed: one color image is used to refine the warped (to both sides) and up-sampled depth images

6. Processing diagram of our system

7. Generation of preliminary high-resolution depth image • Purpose: to generate high-resolution depth image at the left and right sides Calibrated camera parameters

8. Pre-processing of the depth map • 33 median filter • Low resolution to high resolution depth warping • Forward warping with a block size of 55 pixels • Post-processing by morphological operation • Morphological opening

9. Generation of high quality high-resolution depth image • Color-image-guided depth map refinement

10. Binarizing the warped depth image to have foreground/background classification • Color segmentation by mean-shift algorithm • Label each segment as FG or BG • Refine the corresponding depth values according to the segment categorization • Bilateral filtering is performed for the edge parts of the modified depth image

11. Experiments – system configurations • Color image: 640x480 • Depth image: 176x144 (SR4000) • Intel Core 2 Duo Q6600 2.33 GHz, and DDR2 800 4GB

12. Preliminary high-resolution depth image After depth warping Traditional depth warping After morphological operation

13. High-quality high-resolution depth image After correction Mean-shift Incorrect pixels Bilateral filtering

14. A series of generated depth maps

15. Virtual view synthesis based on various depth maps Hole filled Bilateral filtering on edges Bilateral filtering on edges + Gaussian Bilateral filtering

16. Speed performance

17. Conclusion • A hybrid camera system consisting of high-resolution color camera and low-resolution depth camera is proposed • A high-quality high-resolution depth images can be obtained for the left and right eyes – multi-view video plus depth (N=2 channels). • Future work • Combine Kinect device and stereo camera to get better depth image and cheaper cost