Presented at the Electronic Imaging 2003

1. Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Sila Ekmekci and Thomas Sikora Technical University Berlin Presented at the Electronic Imaging 2003

2. Overview • Multiple Description Coding • Multiple State Video Coding & State Recovery • Unbalanced Multiple State Video Coding & Unbalanced Quantization • Problem Definition • Two Approaches • Results • Approach 1, balanced loss probabilities • Approach 1, unbalanced loss probabilities • Approach 2, balanced loss probabilities • Approach 2, unbalanced loss probabilities • Summary & Conclusions Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 2

3. description 1 D1 Decoder 1 Combine D1,D2>D0 reconstructed data MD Coding original data Channel Decoder 2 description 2 D0 D2 Multiple Description Coding (MDC) • MDC is a forward error correction code (increasing error resilience) • two or more descriptions of the source are sent to the receiver • if only one channel is received the signal can be reconstructed with distortion D1 or D2 respectively. • if both descriptions are received a lower distortion D0 is achieved. • MD codes have low delay property (compared to retransmission approaches or long channel codes). Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 3

4. Multiple State Video Coding (by John Apostolopoulos) • video is coded into multiple independently decodable streams each with its own prediction process and state. • if one stream is lost the other one is still decodable (advantage, error resilience). • correctly received stream enables state recovery for the corrupted stream using bidirectional information from past and future frames (advantage). • different packets are sent over different paths (path diversity independent losses). • distance between frames on the same thread increases (disadvantage, decrease in prediction gain). Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 4

5. State Recovery in Multiple State Video Coding Conventional Prediction Multiple State Prediction & State Recovery Sila Ekmekci Technical University Berlin trade off between prediction gain and error resilience Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 5

6. Unbalanced Multiple State Video Coding • Motivation: Network paths with unbalanced bandwidths or unbalanced loss probabilities  descriptions with unbalanced rates. • Problem: How to perform unbalanced rate adaptation between streams? • Methods for Rate Adaptation: • adaptation of frame rate (by John Apostolopoulos) • adaptation of spatial resolution • adaptation of quantization (subject of this work!!) Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 6

7. Unbalanced Quantized Multiple State Video Coding • Rate adaptation is performed based on quantization. • Video is splitted framewise into two threads (odd and even frames) • Each thread is quantizied with quantization stepsizes QP1 and QP2 respectively (yielding the bitrates R1 and R2). • Problems: • PSNR variations (flickers) occur between the two interleaved differently quantized streams at half the frame rate. •  • Flickers larger than 0.5 dB are visually disturbing. •  • quantization rate adaptation can only be used for small rate changes Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 7

8. stream1 ,R1 D1 Multiple State Video Coding p1 Decoder 1 Combine D1,D2>D0 reconstructed video video Channel Decoder 2 p2 D0 stream 2, R2 D2 Problem Definition • Maximize the average PSNR (PSNRavg) and minimize PSNRdiff over all reconstructed frames given the total bitrate RT=R1+R2.and p1, p2 the loss probabilities over the two paths chosen. Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 8

9. Our Approach (Approach 1) • Analyzing the Rate-PSNR performance of Multiple State Video Coding for the given loss probabilities p1 and p2. • Analysis Steps: • H26L Codec (Version 9.0) is modified to support Multiple State Video Coding • 100 frames are splitted into two threads having 50 frames each. • Each thread is coded and decoded with different quantization stepsizes yielding different bitrates R1 and R2 (RT is fixed). • Each frame is assumed to be transmitted in a separate packet. • 20 different loss patterns for each loss probability (0%, 5%, 10% and 20%) are generated and used for decoding the streams. • Lost frames are substituted by interpolations of the previous and next frame from the other stream. Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 9

10. Approach 1, balanced loss probabilities Sila Ekmekci Technical University Berlin • Balanced loss probabilities call for balanced rate allocation. • The penalty of unbalanced bitrate allocation can be as large as 1.5 dB in PSNRavg for 0% loss probability. • Interaction between streams increases with increasing loss probability ( decrease in PSNRdiff). Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 10

11. Approach 1, unbalanced loss probabilities (1) Sila Ekmekci Technical University Berlin • The first channel is lossless: • as p2 increases more bitrate should be shifted to the first channel (more unbalance) • maximum PSNRavg increases with increasing p2 (due to the increased interactions between the streams)  PSNRavg could be increased by discarding the low quality stream and substituting it by an interpolated one at high unbalance. Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 11

12. Approach 1, unbalanced loss probabilities (2) Sila Ekmekci Technical University Berlin • Both channels are lossy: • bitrate should be shifted towards the stream with a smaller loss probability to maximize PSNRavg.(intuitive) • flickers increases with increasing unbalance in bitrates and decreases with increasing number of interactions between streams (less intuitive). Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 12

13. Motivation for Approach 2 Sila Ekmekci Technical University Berlin • Observation for the lossless case: • Using the high rate stream and replacing the lower rate stream by interpolations from the high rate stream increases PSNRavg and decreases PSNRdiff if the rate unbalance exceeds a certain threshold. • Howcan we combine both curves to get the best of both methods? Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 13

14. Approach 2 • There are two reconstruction options for each frame received • 1- using it as received. • 2- discarding and substituting it by its interpolated counterpart from the other stream. • Using Approach 2, we can get even better results than shown on the previous slide (solid curves), since the reconstruction decision is made now on frame base. • Approach 2 is used just for analysis and not for reconstruction. • maximum PSNRavg for various systems at lossless reception (Foreman, RT=140kbps): • single description: 36.1 dB • Approach 1: 35.3 dB • Approach 2: 36.4 dB (more than single description) Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 14

15. Approach 2, balanced loss probabilities Sila Ekmekci Technical University Berlin • unbalancing the rate allocation does not increase PSNRavg (as opposed to the lossless case) • the variations in PSNRavg and PSNRdiff over R1 are much smaller as compared to Approach 1 (less penalty for unbalancing the bitrates). • PSNRdiff decreases over all possible allocations as compared to Approach 1. Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 15

16. Approach 2, unbalanced loss probabilities (1) Sila Ekmekci Technical University Berlin • The first channel is lossless: • maximum achievable PSNRavg is only slightly increased. • PSNRdiff is decreased. • variation ofPSNRavg and PSNRdiff over R1 are smaller compared to Approach 1. Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 16

17. Approach 2, unbalanced loss probabilities (2) Sila Ekmekci Technical University Berlin • Both channels are lossy: • maximum achievable PSNRavg is only slightly increased. • PSNRdiff is decreased. • variations ofPSNRavg and PSNRdiff are smaller compared to Approach 1. • maximum PSNRavg is shifted towards right as compared to Approach 1 (more support for unbalanced operation). Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 17

18. Summary & Conclusion • Two differently quantized descriptions of a video sequence are generated using Multiple State Video Coding. • The descriptions are sent on different channels with independent loss probabilities. • Different quantizations yield flickers after interleaving the streams. • Our objective is to maximize PSNRavg and minimize PSNRdiff . • Two Analysis Approaches for optimal rate allocation are discussed each considering a different reconstruction algorithm at the receiver. • Unbalanced loss probabilities call for unbalanced bitrate allocations. • Unbalanced bitrates can increase PSNRavg even in lossless reception of both streams. • Approach 2 yields smaller flickers due to the increased interactions between streams. • If the lower quality stream can be used to enhance the interpolated images the reconstruction results can be further improved (future work) Sila Ekmekci Technical University Berlin Unbalanced Quantized Multiple Description Video Transmission using Path Diversity Electronic Imaging 2003 18