Error Control and Concealment for Video Communication

1. Error Control and Concealment for Video Communication CMPT820 Summer 2008 Michael Jia

2. Reference “Error Control and Concealment for Video Communication : A Review” YAO WANG, Member, IEEEQIN-FAN ZHU, Member, IEEE

3. Outline • Introduction • Error Detection • Error Concealment at Coder • Error Concealment at Decoder • Interactive Error Concealment • Conclusion

4. Introduction • 2 Types of Transmission Errors • Random Bit Errors • Bit inversion, bit insertion, bit deletion • Erasure Errors • Packet loss, burst errors, system failures • More destructive • Common usage of VLC makes them no differences in video streaming

5. Introduction • Lossless recovery • FEC (Forward Error Correction) • ECC (Error Control Coding) • ARQ (Automatic Retransmission Request) • Not necessary for video transmission, human eyes can tolerate a certain degree of distortion • Focus on signal-reconstruction and error-concealment techniques

6. Introduction • Typical structure of a video communication system

7. Introduction • 3 groups of error-concealment techniques • Forward error concealment • Encoder plays the primary role • Error concealment by post-processing • Decoder fulfills the task • Interactive error concealment • Main concerns • Effectiveness • Required delay • Bit-rate overhead • Processing complexity

8. Outline • Introduction • Error Detection • Error Concealment at Coder • Error Concealment at Decoder • Interactive Error Concealment • Conclusion

9. Error Detection • At transport layer • Adding header information (sequence number) • H.223 • FEC (Forward Error Correction) • H.223, H.261 • More reliable • Need more bandwidth

10. Error Detection • At decoder • Based on characteristics of natural video signals • Pixel value differences of neighboring lines (compare to a threshold) • Differences between boundary pixels in a block and its four neighbor blocks • Obvious false value of quantization step size or DCT coefficients • Insert synchronization code word at the end of line of blocks • No additional bits or very few • Rely on smoothness of signal

11. Outline • Introduction • Error Detection • Error Concealment at Coder • Error Concealment at Decoder • Interactive Error Concealment • Conclusion

12. Error Concealment at Coder (1) • Layered Coding with Transport Prioritization • Most popular and effective (MPEG-2)

13. Error Concealment at Coder (1) • Layered Coding • Base layer – most important layer, with acceptable quality • Transport prioritization • Deliver base layer with higher degree of error protection • High priority channel • Re-transmission and/or FEC • No explicit bit-rate overhead • Complicate structure and coding overhead • H.264 AVC/SVC • Redundant pictures • Data Partitioning

14. Error Concealment at Coder (2) • Multiple-Description Coding (MDC) • Several parallel channels • Independent error events • Small probability of “all channels down” • Multiple “Descriptions” • Several coded bit streams • Transmitted over separate channels • Any one will work

15. Error Concealment at Coder (2) • Multiple-Description Coding (MDC)

16. Error Concealment at Coder (3) • Joint Source and Channel Coding • Source-channel interaction at a lower level • Given channel error characteristics, design quantizer and entropy coder for to minimize the effect of errors • For general sources, noisy channel  coarse quantizer is better • For image signals, noisy channel  • fewer bits to high-frequency coefficients • more bits to low-frequency coefficients

17. Error Concealment at Coder (4) • Robust Waveform Coding • Intentionally keep some redundancy in source-coding • Layered coding and MDC both belong to this category • Adding auxiliary information in waveform coder • MPEG-2: sending motion vectors for microblocks in I-frames • Restricting prediction domain • H.263/H.264: prediction is confined within each slice

18. Error Concealment at Coder (5) • Robust Entropy Coding • Add redundancy in entropy-coding • To help detect bit errors and prevent error propagation • Self-Synchronizing entropy coding • Add a synchronization code word • H.261, H.263, MPEG-4 • Error-Resilient entropy coding (EREC) • MPEG-4 uses RVLC (reversible VLC)

19. Error Concealment at Coder (7) • Transport-Level Control • Add redundancies at transport level • Prioritized transport for layered coding • Robust packetization • Spatial block interleaving • Dual transmission of important information • H.264 AVC/SVC • NAL unit syntax structure • Parameter Sets

20. Error Concealment at Coder (Summary)

21. Outline • Introduction • Error Detection • Error Concealment at Coder • Error Concealment at Decoder • Interactive Error Concealment • Conclusion

22. Error Concealment at Decoder • Preview • Perform error concealment at the decoder • Can be used in conjunction with the auxiliary information provided by the source coder • Low frequency components dominate images of natural scenes • Color values of adjacent pixels vary smoothly except sharp edges • Human eyes tolerate more distortion to high-frequency components

23. Error Concealment at Decoder (1) • Motion-Compensated Temporal Prediction • Replace damaged MB with the motion compensated block • Very effective when all the motion information in the base layer • Widely used (MPEG-2) • What if motion information or coding mode is lost? • Will discuss in next slide

24. Error Concealment at Decoder (2) • Recovery of Motion Vectors and Coding Modes • What if we lost motion vectors or coding modes? • Interpolated from spatially and temporally adjacent blocks • Estimation of coding modes • Simple: treat as ‘intracoded’ • More sophisticate: MPEG-2 (See tables)

25. Error Concealment at Decoder (2) • Estimation of motion vectors • Set to zeros (works well for low motion video) • Use MV of the corresponding block in the previous frame • Use the average of MVs from spatially adjacent blocks • Use the median of MVs from spatially adjacent blocks • Select one of the above methods depending on least boundary matching error

26. Error Concealment at Decoder (3) • Maximally Smooth Recovery • A constrained energy minimization approach • Minimize a measure of spatial and temporal variation between adjacent pixels in this block and its spatially and temporally neighboring blocks • Measure differences in 3 domains • Spatial – adjacent blocks • Temporal – prediction block in previous frame • Frequency – received coefficients for this block

27. Error Concealment at Decoder (3) • Maximally Smooth Recovery • 2 sample spatial smoothness measures

28. Error Concealment at Decoder (4) • Spatial- and Frequency-Domain Interpolation • A coefficient in a damaged block is likely to be close to the corresponding coefficients in spatially adjacent blocks • Interpolate from four neighbor blocks • Maybe not accurate (8-pixel is too far) • Interpolate from four 1-pixel-wide boundaries • 2 pixels in 2 nearest boundaries • 4 pixels in all 4 boundaries • See picture in next slide

29. Error Concealment at Decoder (4) • Spatial- and Frequency-Domain Interpolation • Interpolate from four 1-pixel-wide boundaries

30. Error Concealment at Decoder (Summary)

31. Outline • Introduction • Error Detection • Error Concealment at Coder • Error Concealment at Decoder • Interactive Error Concealment • Conclusion

32. Interactive Error Concealment • Preview • If a backward channel is available, can achieve better performance by cooperation • Based on the feedback • At source coder – coding parameters can be adapted • At transport level – adjust the portion of bandwidth used for error control • Decoding delay issue

33. Interactive Error Concealment (1) • Selective Encoding for Error Concealment • Simple – code next frame in intramode • Error stopped in about one round-trip time • Will cause bit-rate increase • Send identity info back, perform error concealment at the same time • Continue encode without using the affected area • Perform same error concealment procedure (need a prediction frame buffer) • See picture in next slide

34. Interactive Error Concealment (1) • Selective Encoding for Error Concealment

35. Interactive Error Concealment (1) • Selective Encoding for Error Concealment

36. Interactive Error Concealment (1) • H.263 uses more prediction frame buffers (reference picture selection mode)

37. Interactive Error Concealment (2) • Adaptive Transport for Error Concealment • Retransmission is unacceptable for real-time video applications? • Not always • For one-way video broadcast, we may tolerate a few seconds delay • For multipoint video conferencing, use MCU (multipoint control unit) • If retransmission is controlled properly, end-to-end quality can be improved • Both H.323 and H.324 defined such mechanisms

38. Interactive Error Concealment (3) • Retransmission Without Waiting • Wait for the retransmission data • Not good, may freeze the display • Cause transit delay and accumulation delay • Without waiting • Request the retransmission • Conceal the error • Track the affected pixels • Correct them upon the arrival of the retransmission data

39. Interactive Error Concealment (4) • Prioritized, Multi-copy Retransmission • Effective in very lossy channels • Video streaming via PSTN • Are you kidding me? • Send multiple copies of a lost packet • Use layered coding • # of retransmission trials and # of copies are proportional to the importance of the layer

40. Interactive Error Concealment (Summary)

41. Outline • Introduction • Error Detection • Error Concealment at Coder • Error Concealment at Decoder • Interactive Error Concealment • Conclusion

42. Conclusion • Real-time video communication doesn’t require lossless delivery; signal-reconstruction and error-concealment techniques are more effective. • Add redundancy when encoding or delivering • Estimate missing information when decoding • Inform sender what is lost • Burstiness has a significant impact on the choice of algorithms

43. Questions?

44. Thank You