Spatial-Temporal Consistency in Video Disparity Estimation

1. Spatial-Temporal Consistency in Video Disparity Estimation ICASSP 2011 RamsinKhoshabeh, Stanley H. Chan, Truong Q. Nguyen

2. Outline • Introduction • Proposed Method • Image-based disparity map estimation • Temporal consistency • Experimental Result

3. Introduction • Stereo disparity estimation is an integral problem associated with 3D content delivery • Two type of existing algorithm • Local • Global : minimizing energy function • Even applying the best of existing methods to individual frames of stereo sequences yields temporally inconsistent disparity maps (fast, lack the accuracy) (slow)

4. Introduction • The goal is to present a method to generate accurate and spatio-temporally consistentdisparity maps from complex stereo video sequences.

5. Proposed Method • Use Image-basedtechnique • Video disparity problem in space-time is computationally impractical • But we lose the consistency between consecutive frames  noisy • Improve the temporal consistency

6. Proposed MethodImage-based disparity map estimation • In this step, disparity maps are computed for each frame individually. • Use a global method using Hierarchical Belief Propagation (HBP) for inferencing. • Energy function : P : set of pixels in an image L : finite set of labels A labeling f assigns a label fpϵLto each pixel p ϵP Data cost Discontinuity cost How well the labeling fit the node

7. Proposed MethodImage-based disparity map estimation • Discontinuity costenforces the assumption that labels should vary slowly. • Except for significant changes along object boundaries

8. Proposed MethodImage-based disparity map estimation • Data cost is computed over a large window for each pixel using locally adaptive support weights [11] Strength of grouping by proximity Strength of grouping by similarity : Color difference : Spatial distance only points with a high probability of belonging to the same object contribute significantly to the cost calculation [11] K. J. Yoon and I. S. Kweon, “Locally Adaptive Support-Weight Approach for Visual Correspondence Search,” in CVPR, 2005.

9. Proposed MethodImage-based disparity map estimation • use the method of [5] to minimize the energy over the entire image in a coarse-to-fine manner. • Use the hierarchy to reduce the number of message passing iterations . [5] P. Felzenszwalb and D. Huttenlocher, “Efficient Belief Propagation for Early Vision,” in CVPR, 2004, pp. 261–268.

10. Proposed Methodtemporal consistency • Disparity should be a piecewise smooth function in time, except for discontinuities at object borders • Consider the sequence of disparity maps as a space-time volume t y x

11. Proposed Methodtemporal consistency TV-norm Forward difference Add (βx,βy, βt) so that we can control the relative emphasis allows us to handle both spatial and temporal consistency simultaneously, by tuning the parameters (βx, βy, βt) Total variation [14] S. H. Chan, R. Khoshabeh, K. B. Gibson, P. E. Gill, and T. Q. Nguyen,“An augmented lagrangian method for total variation video restoration,” in ICASSP, May 2011

12. Experimental Result Spatial noise Temporally inconsistencies Remove error Preserve object edges

13. Experimental Result

14. Experimental Result • Add Gaussian noise to simulate real sequences • Bad pixel is defines as any pixel that has an estimated disparity that |Dest – Dreal | > threshold(set at 1)

15. Experimental Result Evaluate the efficacy of proposed TV method in improving arbitrary disparity estimates.