Encoder-Based Rate Smoothing and Quality Control for Low Delay Video Coding

1. SPIE VCIP, Jan 17, 2002 Encoder-Based Rate Smoothing and Quality Control for Low Delay Video Coding Zhihai He and Chang Wen Chen Interactive Media Sarnoff Corporation

2. Introduction - Problem • Video coding has two basic modes: • constant bit rate (CBR) coding • variable bit rate (VBR) coding CBR video coding • Low-delay video coding. • Video quality fluctuation due to the time-varying • scene activities in the video sequence.

3. Introduction - Problem MPEG-4 Table-tennis QCIF 96 kbps 15 fps PSNR • Very large PSNR • fluctuation Frame

4. Introduction - Problem VBR Coding • Smooth video presentation • quality • Large rate fluctuation Bits Controlled PSNR: 35 dB From the standpoint of transmission efficiency, network traffic management and resource allocation, bursty VBR video stream is much harder to handle than the CBR video.

5. Introduction - Problem Rate smoothing Video Encoder • Bits stream buffers • on the transmission path • encoder buffer / router buffer / • decoder buffer • Function as low pass filters [Examples] Channel Transmission buffer based rate smoothing • Introduce large transmission delay • Low-pass filtering mechanism is not very effective, • especially for medium/long-term rate fluctuation.

6. Introduction - Problem 3 frames 7 frames 15 frames 30 frames

7. Encoder-Based Rate Smoothing In this work • Encoder-based rate smoothing • The encoder smoothes out the rate shape by itself while • maintaining a controlled video quality. • No additional transmission delay is introduced. How • The encoder estimates the rate-distortion (R-D) function • of the current frame. • Makes a good trade-off between rate and quality (distortion) • based on the estimated R-D functions.

8. Estimation of R-D Functions • R-D Estimation requirements • Fast (small computation delay) and accurate • a prior estimation: before quantization and coding In our previous work [Zhihai He and Sanjit K. Mitra, August CSVT 2001], we have developed a fast and accurate R-D estimation algorithm, which can estimate the R-D functions before quantization and coding.

9. Estimation of R-D Functions [Brief Review] H.263 / MPEG-4 Coding system Coding performance depends on Characteristics of the input source data Capability of the coding algorithm (MPEG-2, H263, MPEG-4, etc) to explore these characteristics  

10. Estimation of R-D Functions Definition Distribution of DCT coefficients q quantization step size  percentage of zeros among the quantized transform coefficients. q Rate R(q) R()  Distortion D(q) D() Observations  monotonically increases with q. There is a one-to-one mapping between them q-domain  -domain

11. Ideas [1]: Characteristic Rate Curves Describe the source data • Classical R-D analysis: variance insufficient Proposed: Characteristic rate curves • Introducing two rate curves in the  -domain: Qz() and Qnz() to characterize the input source data

12. Ideas [2]: Rate Curve Decomposition Model the coding algorithm Fourier analysis f(t) = Ancos(nt) + Bnsin(nt) R(): actual rate curve Rate curve decomposition R() = A() Qz() + B( )Qnz() + C() Qz() and Qnz() : modeling the source data A(), B( ) and C() : modeling the coding algorithm

13. Rate Curve Decomposition A Set of Small and Simple Problems A Large Problem R() = A() Qz() + B( )Qnz() + C() Zeros Qz() Zeros Qz() Input Picture Input Picture Non-zeros Qnz() Zeros Qz() R-D functions R-D functions Coding Algorithm Input Picture

14. Estimation of R-D Functions How does it work? • We first show that the two characteristic rate curves • Qz() and Qnz() have unique properties. • Based on these properties, they can be estimated with • a fast algorithm.

15. Characteristic Rate Curves Sample images Wide range of image characteristics

16. Characteristic Rate Curves Plots of Qnz() : Top Qz(): Bottom • UTQ • Wavelet

17. Characteristic Rate Curves Motion compensated pictures from Foreman MPEG-4 • Very small picture- • dependent variation • Straight line

18. Characteristic Rate Curves The same curves in the q-domain Image-dependent variation (most difficult part.) Highly nonlinear

19. Estimation of R-D Functions R() = A() Qz() + B( )Qnz() + C() • For a given encoder, its decomposition coefficients • A(), B( ) and C()is fixed. • Using summation: R() = A() Qz() + B( )Qnz() + C() • we obtain the rate curve R(), which is then mapped into • q-domain R(q).

20. Estimation of R-D Functions Six test images Coding algorithms: SPIHT Stack-run Estimate their R-D curves before coding.

21. Estimation of R-D Functions For SIPHT Relative estimate error less than 4%.

22. Estimation of R-D Functions For JPEG The first 4 images Relative estimate error less than 5%. Digital camera applications

23. Estimation of R-D Functions Frames: 30, 60,90 and 120 MPEG-4 Carphone 15 fps Accurate frame-level rate control PSNR Bits

24. Encoder-Based Rate Smoothing Trade-off between rate and quality for low-delay video coding. Objective • The output bit rate shape of the video encoder is smoothed, • not bursty. (Just like the transmission buffer rate smoothing) • The output picture quality should be well controlled • or maintained if possible. • Introduce no addition transmission delay.

25. Encoder-Based Rate Smoothing Observations • Good video presentation quality in practice: • Dramatic change of quality from picture to picture is • not acceptable. • However, • Smooth change (temporal variation) of picture quality • within a controlled range is acceptable. [Demo later]

26. Encoder-Based Rate Smoothing Idea How to smooth out the rate shape while maintain good video quality? Find the shortest path Rate: as smooth as possible. Quality: very smooth change with +/- 1dB 35 dB Rate 33 dB Frame time

27. Encoder-Based Rate Smoothing Algorithm Frame n-1: Just coded Bit Rate: Rn-1 If R-- < Rn-1 < R+ if Rn-1 > R+ if Rn-1 < R- Rn-1 R+ R+ Current frame Target 34 dB 34+1 dB: R+ 34-1 dB: R-- Rn =

28. Encoder-Based Rate Smoothing Properties • Rate: as smooth as possible. • Quality: very smooth change with +/- 1dB of the • target quality. • Very effective in smoothing, both short term and long term. • No addition delay 35 33

29. Experimental Results MPEG-4 Foreman Target 34 dB Controlled variation: 1dB

30. Experimental Results PSNR

31. Experimental Results Demo1 Foreman CIF 15 fps Target 34 dB Demo2 NBA CIF 15 fps Target 33 dB Demo3 TexasWild CIF 15 fps Target 32 dB

32. Conclusion • An encoder-based rate smoothing algorithm • has been developed, smooth out the rate shape • while maintaining well-controlled visual quality • For very low delay coding application.