670 likes | 869 Views
Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction. Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering, National Chiao Tung University, Taiwan, R.O.C. Introduction. Transmitted Video. Wireless Network. Error Resilience.
E N D
Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction Chen-Yu Tseng (曾禎宇) Department of Electronic Engineering, National Chiao Tung University, Taiwan, R.O.C.
Introduction Transmitted Video Wireless Network Error Resilience Received Video Packet loss Error Concealment
Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion
Compressed Video Over Wireless Network Packet-Based Video Transmission Encoded Sequence Decoded Sequence P P I I Packet Loss P P P P P P P P Error Propagation time time
Prior Arts of Error Concealment • Block-Level Concealment • Spatial Based • Spatial-Temporal Based • Motion Vector Field Based • Frame-Level Concealment Missing Block Correctly Received Parts
Block-Level Concealment • Spatial Interpolation Based [9-11] Original Sequence Edge Preserving Bilinear Interpolation
Block-Level Concealment • Spatial-Temporal Based [12-15] Missing MB Substitute MB Reference Frame Current Frame Boundary Matching Algorithm (BMA) W. M. Lam et al. [12]
Block-Level Concealment • Motion Vector Field Based [16-17] Recovered MV Missing MB Substitute MB Reference Frame Current Frame Interpolation in MV Field
Whole-Frame Loss Example of 3G Applications: • QCIF Video Transmission at 64 kbps. Frame Rate: 10 fps. Average 800 bytes per frame • General Packet Size: 1kB Packet Loss May Cause Whole-Frame Loss!!
Frame-Level Concealment • Motion Vector Recovery Based on Optical Flow Recovered MV Ref MV Substitute MB Reference Frame Current Missing Frame
Frame-Level Concealment • S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “An error concealment algorithm for streaming video,” Proc. ICIP 2003, 2003 [22] • S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “Concealment of whole-frame loss for wireless low bit-rate video based on multiframe optical flow estimation,” IEEE Trans. Multimedia, vol. 7, no. 2, pp. 316-329, Apr. 2005. [23] • P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni, and F. Rovati, “Frame concealment for H.264/AVC decoders,” IEEE Trans. Consumer Electronics, vlo. 51, no. 1, pp. 227-233, Feb. 2005. [24] • Zhenyu Wu and J. M. Boyce, “An error concealment scheme for entire frame losses based on H.264/AVC,” Proc. ISCAS 2006, May 2006.[25]
Optical Flow (OF) • Motion Consistency t-2 t-2 t-1 t-1 t time Optical Flow t
Motion Recovery Based on Optical Flow Packet lost t-2 time time t-1 Corrupted Optical Flow Original Optical Flow t time Optical Flow
Optical Flow Estimation • General Model of Optical Flow Equation[27] • Motion Vector substitutes for OF (1) (2) (i, j) (iF, jF) Fn-1 Fn Motion Vector Projection
MV Versus OF • MV Block Matching • OF Motion Trajectory Block Matching Reference Frame Foreman #4 Current Frame Foreman #5
Temporal Aliasing Problem Wrong MV Correct MV Projected MV Projected MV t-2 t-2 t-1 t-1 t Missing Frame t Correct MV Projection Wrong MV Projection
Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels
Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels
Pixel-based Concealment • Estimation of optical flow with temporal regularization n n-2 n-1 MV MV MV MV TERM MVH MVH TERM MVH MVH There are other temporal interpolators for the MVH, including median filter and weighted averages decaying overtime. Belfiore et al. [22] find that mean value consistently yields the best result, thus validating the linear velocity assumption.
Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels t t-1
Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels t t-1
Pixel-based vs. Block-Based t t t-1 t-1 Pixel-based Block-based
Reference frame buffer Searching of suitable reference Motion vector projection Statistics collection MB-Level MV field estimation Block-Level MV field estimation Picture reconstruction t t-1 Block-based Concealment • Schematic of P. Baccicht’s Algorithm[24]
t Problems of MV Projection • Conflict State • Non-covered State Non-covered State Conflict State t-1
Problems of MV Projection • Conflict State • Non-covered State Possible Reasons • Wrong Motion Vector • Object Warping, Occlusion, or Non-translation Motion. • Frame Boundary
Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion
Motion Vector Correction • Motion Vector True Motion Trajectory • Property of True Motion Trajectory • Temporal Consistency • Spatial Consistency Temporal Consistency t-2 t-1 t Spatial Consistency
Motion Vector CorrectionBased on Fuzzy Logic Motion Vector Field Original MV: Corrected MV: Reliability of : Spatial Reliability: Temporal Reliability: Spatial Difference Temporal Difference
Temporal Reliability • Temporal Difference t-2 t-2 t-1 t-1 t t time time High temporal difference Low temporal reliability Low temporal difference High temporal reliability 1 0
1 0 Spatial Reliability • Spatial Difference Motion Vector Field Motion Vector Field
Motion Vector CorrectionBased on Fuzzy Logic y x Motion Vector Field Original Frame Original MV Field Corrected MV Field
MV Projection with MV Correction y x Original MV Field Corrected MV Field t t t-1 t-1
Proposed MV Projection • Why Concealment? Release Error Propagation Can We Skip from Error Frame? P P I I P P P P P P P P Error Propagation Error Frame Error Frame time time
MV Projection Forward vs. Backward Forward MV Projection Packet lost Packet lost Backward MV Projection time time time Corrupted Optical Flow Original Optical Flow Corrupted Optical Flow
MV Projection Forward vs. Backward OF OF Reference Frame Reference Frame Missing Frame Missing Frame Backward projected OF Forward projected OF OF (a) Forward motion projection (b)Backward motion projection
MV Projection Forward vs. Backward Backward Projected MV Missing frame Missing frame (i, j) (iF, jF) (iB, jB) Fn-1 Fn (i, j) Fn-1 Fn Fn+1
Advantage ofBackward MV Projection • Easy to implement. • Avoid from conflict or non-cover states. • Invariance of I-frame position. Missing frame Missing frame P P I I P P P time P P time P P P Missing frame No MV Missing frame P P P P I I P time P P time P P P Forward Motion Projection Backward Motion Projection
Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion
Schematic of Proposed System Wireless Network Encoder with MV Correction Decoder with Missing Frame Prediction Error Resilience Error Concealment
Motion Vector Correction • Spatial-Temporal MV Correction (3-D) Iterative Correction time … n-2 MV Field n x n-1 The number of MV correction iteration effects the result of concealment. … y The PSNR is the concealment result (foreman FLR10) with different number of MV correction iteration
…. …. …. …. …. …. MVn-2 MVn-2 MVn-2 MVn-2 MVn-2 MVn-2 MVn-1 MVn-1 MVn-1 MVn-1 MVn-1 MVn-1 MVn MVn MVn MVn MVn MVn MVn+1 MVn+1 MVn+1 MVn+1 MVn+1 MVn+1 MVn+2 MVn+2 MVn+2 MVn+2 MVn+2 MVn+2 …. …. …. …. …. …. Proposed 2-StepMotion Vector Correction Forward MV correction Backward MV correction Iteration time
Concealment PSNRwith Different MV Correction The PSNR is the concealment result (foreman FLR10) 2-Step: 33.4666 dB Bidirectional: 33.1826 dB
Motion Compensation PSNR PSNR Drop With MV Correction : 32.0382 dB Without MV Correction: 34.5538dB (foreman)
PSNR Drop Effect: Increase the bit rate Reason: Inconsistent Motion MV correction is based on motion consistency. In this situation, concealment is also hard to work. We preserve the original MV to avoid MC PSNR drop.
PSNR Drop Control Factor • If Residual_MADcorrect > • f * Residual_MADuncorrect • MVcorrect = MVoriginal • Residualcorrect = Residualoriginal ME MC MV Correction MC f is the control factor
MC_PSNR vs. EC_PSNR Foreman QCIF MC_PSNR: Motion Compensation PSNR EC_PSNR: Error Concealment PSNR (Backward MV concealment FLR10)
MC_PSNR vs. EC_PSNR Foreman QCIF MC_PSNR: Motion Compensation PSNR EC_PSNR: Error Concealment PSNR (Backward MV concealment FLR10)
Error Concealment Results • Foreman QCIF Backward Concealment FLR10 (#63) MAD Factor = inf MAD Factor = 2.75
Refinement of Concealment • MV Temporal Consistency Check t-2 t • Frame Spatial Consistency Check
MV Temporal Consistency Check t-2 t Difft(i, j) = || MVt(i, j) – MVt-2(iB, jB) || ,Where (iB, jB) = (i, j) + 2*MVt(i, j) If Difft(i, j) > Diff_temporal_TH MVt(i, j) = MV’t(i, j) , if Diff_v(i, j) < Diff_h(i, j), MV’t(i, j) = ( MVt(i-1, j) + MVt(i+1, j) )/2 else MV’t(i, j) = ( MVt(i, j-1) + MVt(i, j+1) )/2 where Diff_v(i, j) = || MVt(i-1, j) - MVt(i+1, j) || and Diff_h(i, j) = || MVt(i, j-1) - MVt(i, j+1) || MV Refinement