1 / 67

Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering, National Chiao Tung University,

Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction. Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering, National Chiao Tung University, Taiwan, R.O.C. Introduction. Transmitted Video. Wireless Network. Error Resilience.

kishi
Download Presentation

Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering, National Chiao Tung University,

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction Chen-Yu Tseng (曾禎宇) Department of Electronic Engineering, National Chiao Tung University, Taiwan, R.O.C.

  2. Introduction Transmitted Video Wireless Network Error Resilience Received Video Packet loss Error Concealment

  3. Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion

  4. Compressed Video Over Wireless Network Packet-Based Video Transmission Encoded Sequence Decoded Sequence P P I I Packet Loss P P P P P P P P Error Propagation time time

  5. Prior Arts of Error Concealment • Block-Level Concealment • Spatial Based • Spatial-Temporal Based • Motion Vector Field Based • Frame-Level Concealment Missing Block Correctly Received Parts

  6. Block-Level Concealment • Spatial Interpolation Based [9-11] Original Sequence Edge Preserving Bilinear Interpolation

  7. Block-Level Concealment • Spatial-Temporal Based [12-15] Missing MB Substitute MB Reference Frame Current Frame Boundary Matching Algorithm (BMA) W. M. Lam et al. [12]

  8. Block-Level Concealment • Motion Vector Field Based [16-17] Recovered MV Missing MB Substitute MB Reference Frame Current Frame Interpolation in MV Field

  9. Whole-Frame Loss Example of 3G Applications: • QCIF Video Transmission at 64 kbps. Frame Rate: 10 fps. Average 800 bytes per frame • General Packet Size: 1kB Packet Loss May Cause Whole-Frame Loss!!

  10. Frame-Level Concealment • Motion Vector Recovery Based on Optical Flow Recovered MV Ref MV Substitute MB Reference Frame Current Missing Frame

  11. Frame-Level Concealment • S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “An error concealment algorithm for streaming video,” Proc. ICIP 2003, 2003 [22] • S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “Concealment of whole-frame loss for wireless low bit-rate video based on multiframe optical flow estimation,” IEEE Trans. Multimedia, vol. 7, no. 2, pp. 316-329, Apr. 2005. [23] • P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni, and F. Rovati, “Frame concealment for H.264/AVC decoders,” IEEE Trans. Consumer Electronics, vlo. 51, no. 1, pp. 227-233, Feb. 2005. [24] • Zhenyu Wu and J. M. Boyce, “An error concealment scheme for entire frame losses based on H.264/AVC,” Proc. ISCAS 2006, May 2006.[25]

  12. Optical Flow (OF) • Motion Consistency t-2 t-2 t-1 t-1 t time Optical Flow t

  13. Motion Recovery Based on Optical Flow Packet lost t-2 time time t-1 Corrupted Optical Flow Original Optical Flow t time Optical Flow

  14. Optical Flow Estimation • General Model of Optical Flow Equation[27] • Motion Vector substitutes for OF (1) (2) (i, j) (iF, jF) Fn-1 Fn Motion Vector Projection

  15. MV Versus OF • MV  Block Matching • OF  Motion Trajectory Block Matching Reference Frame Foreman #4 Current Frame Foreman #5

  16. Temporal Aliasing Problem Wrong MV Correct MV Projected MV Projected MV t-2 t-2 t-1 t-1 t Missing Frame t Correct MV Projection Wrong MV Projection

  17. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels

  18. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels

  19. Pixel-based Concealment • Estimation of optical flow with temporal regularization n n-2 n-1 MV MV MV MV TERM MVH MVH TERM MVH MVH There are other temporal interpolators for the MVH, including median filter and weighted averages decaying overtime. Belfiore et al. [22] find that mean value consistently yields the best result, thus validating the linear velocity assumption.

  20. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels t t-1

  21. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels t t-1

  22. Pixel-based vs. Block-Based t t t-1 t-1 Pixel-based Block-based

  23. Reference frame buffer Searching of suitable reference Motion vector projection Statistics collection MB-Level MV field estimation Block-Level MV field estimation Picture reconstruction t t-1 Block-based Concealment • Schematic of P. Baccicht’s Algorithm[24]

  24. t Problems of MV Projection • Conflict State • Non-covered State Non-covered State Conflict State t-1

  25. Problems of MV Projection • Conflict State • Non-covered State Possible Reasons • Wrong Motion Vector • Object Warping, Occlusion, or Non-translation Motion. • Frame Boundary

  26. Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion

  27. Motion Vector Correction • Motion Vector  True Motion Trajectory • Property of True Motion Trajectory • Temporal Consistency • Spatial Consistency Temporal Consistency t-2 t-1 t Spatial Consistency

  28. Motion Vector CorrectionBased on Fuzzy Logic Motion Vector Field Original MV: Corrected MV: Reliability of : Spatial Reliability: Temporal Reliability: Spatial Difference Temporal Difference

  29. Temporal Reliability • Temporal Difference t-2 t-2 t-1 t-1 t t time time High temporal difference Low temporal reliability Low temporal difference High temporal reliability 1 0

  30. 1 0 Spatial Reliability • Spatial Difference Motion Vector Field Motion Vector Field

  31. Motion Vector CorrectionBased on Fuzzy Logic y x Motion Vector Field Original Frame Original MV Field Corrected MV Field

  32. MV Projection with MV Correction y x Original MV Field Corrected MV Field t t t-1 t-1

  33. Proposed MV Projection • Why Concealment? Release Error Propagation Can We Skip from Error Frame? P P I I P P P P P P P P Error Propagation Error Frame Error Frame time time

  34. MV Projection Forward vs. Backward Forward MV Projection Packet lost Packet lost Backward MV Projection time time time Corrupted Optical Flow Original Optical Flow Corrupted Optical Flow

  35. MV Projection Forward vs. Backward OF OF Reference Frame Reference Frame Missing Frame Missing Frame Backward projected OF Forward projected OF OF (a) Forward motion projection (b)Backward motion projection

  36. MV Projection Forward vs. Backward Backward Projected MV Missing frame Missing frame (i, j) (iF, jF) (iB, jB) Fn-1 Fn (i, j) Fn-1 Fn Fn+1

  37. Advantage ofBackward MV Projection • Easy to implement. • Avoid from conflict or non-cover states. • Invariance of I-frame position. Missing frame Missing frame P P I I P P P time P P time P P P Missing frame No MV Missing frame P P P P I I P time P P time P P P Forward Motion Projection Backward Motion Projection

  38. Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion

  39. Schematic of Proposed System Wireless Network Encoder with MV Correction Decoder with Missing Frame Prediction Error Resilience Error Concealment

  40. Motion Vector Correction • Spatial-Temporal MV Correction (3-D) Iterative Correction time … n-2 MV Field n x n-1 The number of MV correction iteration effects the result of concealment. … y The PSNR is the concealment result (foreman FLR10) with different number of MV correction iteration

  41. …. …. …. …. …. …. MVn-2 MVn-2 MVn-2 MVn-2 MVn-2 MVn-2 MVn-1 MVn-1 MVn-1 MVn-1 MVn-1 MVn-1 MVn MVn MVn MVn MVn MVn MVn+1 MVn+1 MVn+1 MVn+1 MVn+1 MVn+1 MVn+2 MVn+2 MVn+2 MVn+2 MVn+2 MVn+2 …. …. …. …. …. …. Proposed 2-StepMotion Vector Correction Forward MV correction Backward MV correction Iteration time

  42. Concealment PSNRwith Different MV Correction The PSNR is the concealment result (foreman FLR10) 2-Step: 33.4666 dB Bidirectional: 33.1826 dB

  43. Motion Compensation PSNR PSNR Drop With MV Correction : 32.0382 dB Without MV Correction: 34.5538dB (foreman)

  44. PSNR Drop Effect: Increase the bit rate Reason: Inconsistent Motion MV correction is based on motion consistency. In this situation, concealment is also hard to work. We preserve the original MV to avoid MC PSNR drop.

  45. PSNR Drop Control Factor • If Residual_MADcorrect > • f * Residual_MADuncorrect • MVcorrect = MVoriginal • Residualcorrect = Residualoriginal ME MC MV Correction MC f is the control factor

  46. MC_PSNR vs. EC_PSNR Foreman QCIF MC_PSNR: Motion Compensation PSNR EC_PSNR: Error Concealment PSNR (Backward MV concealment FLR10)

  47. MC_PSNR vs. EC_PSNR Foreman QCIF MC_PSNR: Motion Compensation PSNR EC_PSNR: Error Concealment PSNR (Backward MV concealment FLR10)

  48. Error Concealment Results • Foreman QCIF Backward Concealment FLR10 (#63) MAD Factor = inf MAD Factor = 2.75

  49. Refinement of Concealment • MV Temporal Consistency Check t-2 t • Frame Spatial Consistency Check

  50. MV Temporal Consistency Check t-2 t Difft(i, j) = || MVt(i, j) – MVt-2(iB, jB) || ,Where (iB, jB) = (i, j) + 2*MVt(i, j) If Difft(i, j) > Diff_temporal_TH MVt(i, j) = MV’t(i, j) , if Diff_v(i, j) < Diff_h(i, j), MV’t(i, j) = ( MVt(i-1, j) + MVt(i+1, j) )/2 else MV’t(i, j) = ( MVt(i, j-1) + MVt(i, j+1) )/2 where Diff_v(i, j) = || MVt(i-1, j) - MVt(i+1, j) || and Diff_h(i, j) = || MVt(i, j-1) - MVt(i, j+1) || MV Refinement

More Related