Error-Resilient Coding and Decoding Strategies for Video Communication

1. Error-Resilient Coding and Decoding Strategies for Video Communication Thomas Stockhammer and Waqar Zia Presented by Li Ma

2. Background & Motivation • Video becoming more popular • Advances in bandwidth, capacity enhancements • Requirements: • data transmission rate • Real-time delivery of multimedia data • Limitation: • QoS available is not sufficient to guarantee error-free delivery for all receivers • Motivation: • Provide means of dealing with various transmission impairments

3. Content • Focus on MCP-coded video • Concentrate on tools and features integrated in standard H.264/AVC • Focus on specific tools for improved error resilience • Other advanced error-resilience features not covered: • Multiple description coding • Distributed video coding • Combinations with network prioritization and FEC

4. Outline • Video Communication Systems • Error-Resilient Video Transmission • Resynchronization and Error Concealment • Error Mitigation • Summary

5. Video Communication Systems • End-to-End Video Transmission

6. Video Applications • wide variety of applications • Different bit rate ranges: • HDTV: 20 Mbit/s • MMS on cell: 20 Kbit/s • Different tolerable end-to-end delay • Conversational applications constraints: ≤ 200-250 ms

7. Transmission Impairments • Differences of errors • Wireless networks: • Fading and interference cause burst errors: multiple lost bits • IP network: • Congestion results in packets lost • Methods • Detect presence of errors • Intermediate protocol layers (UDP) could drop erroneous pkts • Video data pkts treated as lost if delayed more than threshold

8. Data Losses in MCP-Coded Video • Transmitted over error-prone channels • Error concealment • Error propagation / Spatiotemporal error propagation

9. Example of Error Propagation • Pt @ t=0 is lost. • Error propagation till t=8 • Intra-coded image transmitted @ t=9

10. Therefore, when data units might get lost, a video coding system should provide: • Means that allow completely avoiding transmission errors • Features that allow minimizing the visual effects of errors • Features to limit spatial and spatiotemporal error propagation

11. Outline • Video Communication Systems • Error-Resilient Video Transmission • Resynchronization and Error Concealment • Error Mitigation • Summary

12. Error-resilient Video Transmission • System Overview

13. Features • @Sender • MBs are grouped in data units and entropy coding used • Error Control before transmission over lossy channel • Forward Error Correction(FEC) • Backward Error Correction (BEC) • Prioritization Methods • Combinations of above • @Receiver • Erroneous and missing data detected and localized • Decoder gets correct data units or error indication • Error concealment applied at positions where no data received • Report loss of data units to encoder

14. Design Principles • error-resilience tools decrease compression efficiency • Main goal: • Shannon’s separation principle: compression separated with transport • In low delay situations, error-free transport is impossible • System Design Principles • 1. Loss correction below codec layer • 2. Error detection • 3. Prioritization methods • 4. Error recovery and concealment • 5. Encoder-decoder mismatch avoidance

15. Video Compression Tools Related to Error Resilience • Slice Structured Coding • Flexible MB Ordering • Scalable Coding • Data Partitioning • Flexible Reference Frame • Intra Information Coding • Pictures Switching

16. Slice Structured Coding • Slices provide spatially distinct resynchronization points within the data for a single frame • Several MBs grouped together: a slice header • Variable sized data units • Encoder can select the location of sync. Points • Motion vector prediction not allowed over boundaries • Encoder decides either: • Allocate fixed number of MB to one slice • Or fixed bits to one slice (matched to pkt size in network)

17. Flexible MB Ordering • Flexible Macroblock Ordering (FMO) • Allows mapping of MBs to Slice groups • A slice group may contain several slices • MBs can be transmitted in flexible and efficient way • Spatially collocated images areas can be interleaved in different slices  greater probability of concealing lost MB • Protection: Can map ROI (region of interest) into a separated slice group

18. Data Partitioning • Loss of some syntax elements of a bit stream results in larger degradation of quality compared to others • E.g. Loss of motion vector • Data partitioning results in Graceful Degradation • Categorize syntax elements • Header information • Motion information • Texture information

19. Layout of compressed data Without Data Partitioning: With Data Partitioning: 2 additional sync. Points available

20. Data Partitioning (Cont.) • Unequal Error Protection (UEP) • Protect partitions of different importance • More important data offered more protection

21. Flexible Reference Frame • H.263 v.1 & MPEG-2 allow only a single reference frame for predicting P frame and mostly 2 for B frame. • Possible to have significant statistical dependencies between other pictures too • Use more frames than just the recent one as reference • Advantages: • Increased compression efficiency • Improved error resilience

22. Example of Flexible Reference Frame

23. Example of Flexible Reference Frame • Enable Subsequences • Use a subsequence of “anchor frames” at lower frame rate • E.g “ P ” • Other frames inserted in between to achieve overall frame rate • E.g “ P’ ” • Error propagation: • Only till next P received

24. Outline • Video Communication Systems • Error-Resilient Video Transmission • Resynchronization and Error Concealment • Error Mitigation • Summary

25. Resynchronization and Error Concealment • Video Packetization Modes • Without FMO(flexible macroblock ordering) • 1. a constant number of MBs within one slice (arbitrary size) • 2. the slice size bounded to some max bytes (arbitrary # of MBs)

26. Video Packetization Modes (Cont.) • With FMO (more flexible) • Slice interleaving • Dispersed MB allocation using checkerboard patterns • Subpictures within a picture • etc.

27. Error Concealment • Basic Idea • Decoder should generate a representation for lost area • Match as close as possible to the lost info • Within manageable complexity • Techniques • Spatial Error Concealment • Temporal Error Concealment • Hybrid Concealment • Other Techniques

28. Spatial Error Concealment • Based on assumption of continuity of natural scene content in space • Use pixel values of surrounding available MBs • Estimate of lost pixel: • αβγ are weighing factors • Determine relative impact of vertical, Horizontal, upper, lower… • Disadvantage • Blurred reconstruction

29. Temporal Error Concealment • Rely on the continuity of a video sequence in time • Use temporally neighboring areas to conceal lost regions • Previous Frame Concealment (PFC) • Use previous corresponding data to copy to current frame • Only good when little motion • Widely used due to simplicity

30. Hybrid Concealment • When only apply spatial concealment • Concealed regions are significantly blurred • When only use temporal error concealment • Significantly discontinuities in the concealed regions • Hybrid temporal-spatial technique applied • MB mode info of reliable and concealed neighbors decide which concealment method to use

31. Hybrid (cont.) • For intra-coded images • Only use spatial concealment • For inter-coded images • Use temporal concealment when more than half of the available neighbor MBs are inter-coded • Otherwise, use spatial concealment • Referred to as Adaptive temporal and spatial Error Concealment (AEC)

32. Selected Results • Performance of different error concealment strategies

33. Selected Performance Results for Wireless • Low-delay and low-complexity requirements • Max allowed buffering at encoder limited to 250ms

34. Using a smaller slice size of 150 bytes is lower in PSNR when error free • Because: • increased packetization overhead • prediction limitations on slice boundaries • Performs good when in lossy channel • Because the loss affects only a small area of the image for fixed slice size

35. Outline • Video Communication Systems • Error-Resilient Video Transmission • Resynchronization and Error Concealment • Error Mitigation • Summary

36. Motivation • Error propagation is major problem over lossy channels • Encoder can change encoding behavior when he finds it’s likely to be lossy or knows decoder suffering losses

37. Operational Encoder Control • Encoder appropriately select parameters • Motion vectors • MB modes • Quantization parameters • Reference frames • Spatial and temporal resolution

39. Conclusion • Bad decisions at the encoder can lead to • Poor results in coding efficiency • Poor in error resilience • Or both • If no feedback is available • an increased percentage of intra MBs performs best • If feedback available • Interactive Error Control is best

40. Outline • Video Communication Systems • Error-Resilient Video Transmission • Resynchronization and Error Concealment • Error Mitigation • Summary

41. Summary • Important to understand video can benefit significantly when data delivered reliably • Introduced error-resilience tools and impact • For good overall performance, should take into account: • the selection of error-resilience tools • rate-distortion-optimized mode selection • the channel characteristics