Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement

1. Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement Takuya Matsuo, Norishige Fukushima and Yutaka Ishibashi VISAPP 2013 International Conference on Computer Vision Theory and Application

2. Outline • Introduction • Related Works • Proposed Method • Weighted Joint Bilateral Filter • Slope Depth Compensation Filter • Experimental Results • Conclusion

3. Introduction

4. Introduction • Goal : Using two filters to get more accurate disparity map in real-time. • Consideration • Noise reduction • Correct edges • Blurring control Goal

5. Related Works

6. Related Works • Stereo Matching Left Image Right Image

7. Related Works

8. Related Works • Flow Chart (Local) • Disparity Map Refinement

9. Related Works • Depth map refinement with filter • Median filter • Bilateral filter Filter Input depth map Output depth map

10. Related Works • Bilateral filter • Space weight:Near pixels has large weight • Color weight:Similar color pixels has large weight • Smoothing • Keep edges • Weak in spike noise

11. Related Works • Joint bilateral filter • Add in the reference image • Color weight is calculated by the reference • Keep object edges of the reference Reference : Low noise Target : High noise Filtered image

12. Related Works • Joint bilateral filter • Noise reduction O • Correct edge O • Blurring X • Mixed depth values • Spreading error regions • Multilateral filter • Space + Color + Depth • Boundary recovering X

13. Proposed Method

14. Proposed Method • Weighted joint bilateral filter • Noise reduction • Edge correction • Slope depth compensation filter • Blurring control

15. Weighted Joint Bilateral Filter • 𝐷: Depth value • 𝑝: Coordinate of current pixel • 𝑠: Coordinate of support pixel • 𝑁: Aggregation set of support pixel • 𝑤(),𝑐(): Space/color weight • 𝜎𝑠,𝜎𝑐: Space/color Gaussian distribution • 𝑅𝑠: Weight map

16. Weighted Joint Bilateral Filter • Add in the weight map • Controlling amount of influence on a pixel • Weight of the edge and error is small Joint bilateral filter - Mixed depth values - Spreading error regions

17. Weighted Joint Bilateral Filter • Making weight map • Space/color/disparity weight • Sum of nearness of space, color, and disparity between center pixel and surrounding pixels.

18. Weighted Joint Bilateral Filter • Mask image is made by Speckle Filter • Detecting speckle noise • Weight of speckle region is 0 Red region: speckle noise Weight = 0

19. Weighted Joint Bilateral Filter

20. Slope Depth Compensation Filter • Weighted joint bilateral filter • Remaining small blurring • Difference between foreground and background color is small • Slope depth compensation filter • Reason of blurring is mixed depth value • Convert mixed value to non-blurred candidate using initial depth map Removing remaining blur

21. Slope Depth Compensation Filter • X in Dx∈ {INITIAL;WJBF;SDCF}

22. Slope Depth Compensation Filter

23. Proposed Method

24. Experimental Results

25. Experimental Results • Evaluating accuracy improvement for various types of depth maps • Block Matching (BM) • Semi-Global Matching (SGM) • Efficient Large-Scale (ELAS) • Dynamic Programing (DP) • Double Belief Propagation (DBP)

26. Experimental Results

27. Experimental Results

31. Experimental Results • Comparing proposed method with cost volume refinement(Teddy). 32 times slower Yang, Q., Wang, L., and Ahuja, N. A constantspace belief propagation algorithm for stereo matching. In Computer Vision and Pattern Recognition(2010).

32. Experimental Results

33. Experimental Results • Device : Intel Core i7-920 2.93GHz • Comparing running time (ms) of BM plus proposed filter with selected stereo methods.

34. Experimental Results

35. Experimental Results • Use the proposed filter for depth maps from Microsoft Kinect.

37. Conclusion

38. Conclusion Contribution • The proposed methods can reduce depth noise and correct object boundary edge without blurring. • Amount of improvement is large when an input depth map is not accurate. Future Works • Investigating dependencies of input natural images and depth maps.