Deblocking Algorithms in Video and Image Compression Coding

1. Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof. Jian-Jung Ding Digital Image and Signal Processing Lab GICE, National Taiwan University

2. Outline • Introduction • Observations of Blocking Artifact • Four Categories of Deblocking Algorithms • Block-Edge Impairment Metric • Comparison • Simulation • Conclusions and Future Work • References Deblocking Algorithms in Video and Image Compression Coding

3. Introduction • What is Blocking Artifacts ? • Discontinuity over the block boundaries • Result from coarse quantization Original Image Highly Compressed Image Deblocking Algorithms in Video and Image Compression Coding

4. A Simple Coarse Quantization Example DCT Q/Q-1 IDCT Deblocking Algorithms in Video and Image Compression Coding

5. Observations of Blocking Artifacts • Two observations of blocking artifact in block-based transform coding • The blocking artifact are more noticeable in flat areas than in complex areas • The motion compensation propagates the blocking artifacts into the next frames Deblocking Algorithms in Video and Image Compression Coding

6. Four Categories of Deblocking Algorithms • In-loop Deblocking Filter • Post-processing Deblocking Algorithms • Pre-processing Algorithms • Overlapping Block Methods Deblocking Algorithms in Video and Image Compression Coding

7. In-loop Deblocking Filter • Mainly used in video compression coding • Add deblocking filtering to the coding loop • Not compatible with the existing standards • Case Study • H.264/AVC • Optimal Post-Process/In-Loop Filtering Deblocking Algorithms in Video and Image Compression Coding

8. Video Source Intra Inter Transform Quantization Coefficient Scanning Bitstream Motion Estimation Motion Compensation Inverse Quantization Entropy Coding Frame Buffer Intra Frame Prediction Inverse Transform In-Loop Filter Motion Vector H.264/AVC In-loop Filter (1/2) In-Loop Filter Deblocking Algorithms in Video and Image Compression Coding

9. H.264/AVC In-loop Filter (2/2) • H.264/AVC encoder must determine the boundary strength (BS) before filtering. • Apply the filters with different strength based on the boundary strength q0 q1 q2 q3 p3 p2 p1 p0 Deblocking Algorithms in Video and Image Compression Coding

10. Buffer Fullness Coder Control Optimal Post-process/In-loop Filtering (1/3) Residue Video Source QP DCT Coefficients Transform Quantization Entropy Coding Inverse Quantization Data out Enable Prediction Inverse Transform Multiplex Buffer Motion Compensation Frame Buffer OPF 0 ILOF Motion Vector Filter with G Previous Decoded Frame Motion Estimation Find Optimal Filter Parameters for Kernel G Deblocking Algorithms in Video and Image Compression Coding

11. Optimal Post-process/In-loop Filtering (2/3) Input Frame with size W× H Filter Kernel with size l × l Reconstructed Frame with size W × H The relationship between , and can be expressed as The optimal filter coefficient matrix must satisfy Row-stacked form of Deblocking Algorithms in Video and Image Compression Coding

12. Optimal Post-process/In-loop Filtering (3/3) Row-stacked form of l Window Matrix ^ I(i,j) l Solve this equation, then we can obtain the optimal filter coefficients G Now the equation can be expressed as Multiply both side of with , we can obtain Deblocking Algorithms in Video and Image Compression Coding

13. Post-processing Deblocking Algorithms • The most popular deblocking methods • Improve the quality of the compressed image and video without the source data • Case Study • Reduction of Blocking Artifacts in DCT Domain • Deblocking Using Weighted Sums of Symmetrically Aligned Pixels • Deblocking Using Offset and Shift Technique Deblocking Algorithms in Video and Image Compression Coding

14. Reduction of Blocking Artifacts in DCT Domain (1/2) DCT-domain filtering where Low Activity High Activity Deblocking Algorithms in Video and Image Compression Coding

15. Reduction of Blocking Artifacts in DCT Domain (2/2) • The neighboring DCT coefficients at the same frequency do not vary radically within a small range • Apply lowpass filter to each frequency to filter the high frequency components resulting from blocking effect Deblocking Algorithms in Video and Image Compression Coding

16. Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (1/4) • Deblocking Frame Deblocking Algorithms in Video and Image Compression Coding

17. Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (2/4) The filtered pixel where the filter coefficients are Deblocking Algorithms in Video and Image Compression Coding

18. Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (3/4) Linear Solution Quadratic Solution where Deblocking Algorithms in Video and Image Compression Coding

19. Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (4/4) Linear Solution Quadratic Solution Deblocking Algorithms in Video and Image Compression Coding

20. Deblocking Using Offset and Shift Technique (1/3) • Estimate the block of activity Horizontal Activity Block boundary Vertical Activity Deblocking Algorithms in Video and Image Compression Coding

21. Deblocking Using Offset and Shift Technique (2/3) • Filtering for Uniform Deblocking Blocks Block boundary offset p1 p2 p3 p4 p5 p6 Block boundary Offset/2 Offset/4 Offset/8 Offset/8 Offset/4 Offset/2 p1 p2 p3 p4 p5 p6 Deblocking Algorithms in Video and Image Compression Coding

22. Deblocking Using Offset and Shift Technique (3/3) • Filtering for Directional Deblocking Blocks • Filtering for Complex Deblocking Blocks Deblocking Algorithms in Video and Image Compression Coding

23. Pre-processing Algorithms (1/2) • Achieve deblocking and enhance the coding efficiency q3 q2 q1 q0 p0 Encoding Decoding p1 p2 p3 Encoding Decoding Deblocking Algorithms in Video and Image Compression Coding

24. Pre-processing Algorithms (2/2) • Our objective is to minimize the difference between I and V and obtain the coefficients Original (I) Solve this equation, then we can obtain the optimal filter coefficients Pre-processed (R) Filtered (V) The difference between I and V Taking derivative Deblocking Algorithms in Video and Image Compression Coding

25. Overlapped Block Methods • Source of Blocking Artifact • Block-based transform does not take the correlation across the block boundary • Motion Estimation in video coding will propagate the blocking artifact to the next frame • How to solve these two problems • Lapped Orthogonal Transform • Overlapped Block Motion Compensation Deblocking Algorithms in Video and Image Compression Coding

26. Lapped Orthogonal Transform(1/3) • The blocks overlap slightly and the redundant information is transmitted for the samples in the block boundaries • Reduce the blocking artifact Transform Coefficients Assume the 1D discrete time signal is a sequence of MNsamples Segments the signal into M blocks and each block with size N Deblocking Algorithms in Video and Image Compression Coding

27. Lapped Orthogonal Transform(2/3) Energy Compaction Measurement Where is the i-th diagonal entry of the matrix where Deblocking Algorithms in Video and Image Compression Coding

28. Lapped Orthogonal Transform(3/3) 1/2 2 0 0 0 0 1/2 1 1 2 2 LOT of Block 1 Block 1 - E 2 2 4 4 DCT 1/2 3 3 6 6 O DCT - 4 4 1 1 Y(0) 1/2 5 5 3 3 Y(2) LOT of Block 2 - Block 2 E 6 6 5 5 Y(4) DCT 1/2 7 7 7 7 Y(6) O - … Y(1) … Y(3) 1/2 Y(5) Y(7) DCT 1/2 - E DCT 1/2 - - O - - - 2 1/2 - - LOT of Block M - Block M - - Deblocking Algorithms in Video and Image Compression Coding

29. Overlapped Block Motion Compensation (1/2) Reference blocks Window Function Reference Frame Overlapped Block Evaluation Target Block Current Frame Motion Vector Output Deblocking Algorithms in Video and Image Compression Coding

30. Overlapped Block Motion Compensation (2/2) Motion Vectors Reference Frame Overlapped Block Window Function Prediction Frame Deblocking Algorithms in Video and Image Compression Coding

31. Block-Edge Impairment Metric (1/2) • The disadvantages of PSNR • It does not always reveal the real quality perceived by the HVS • It can not real the blockiness • Several block-edge impairment metrics have been proposed • GBIM • It can reveal the real blockiness and evaluate the performance of deblocking algorithms Deblocking Algorithms in Video and Image Compression Coding

32. Block-Edge Impairment Metric (2/2) Reconstructed Image after deblocking Reconstructed Image =3.11 =1.05 PSNR=21.75 dB PSNR=21.86 dB Deblocking Algorithms in Video and Image Compression Coding

33. Comparison (1/2) • In-loop Filter • Achieve better improvement because it can refer to the new input video frame • Not compatible with the existing standards • Post-processing Deblocking Algorithms • Good potential to be integrated into existing standards • Blind to the target image and video Deblocking Algorithms in Video and Image Compression Coding

34. Comparison (2/2) • Pre-processing Deblocking Algorithms • Modify the source image and video in advance and reduces the bit rate • Achieve the quality close to the direct compressed image and video • Overlapped Block Methods • Prevents the blocking artifacts from happening in advanced Deblocking Algorithms in Video and Image Compression Coding

35. Simulation (1/3) Original Image Highly Compressed Image DCT Domain Offset and Shift WSSAP Deblocking Algorithms in Video and Image Compression Coding

36. Simulation (2/3) LOT with Q=10 • LOT and DCT PSNR under different quantization strength DCT with Q=10 Deblocking Algorithms in Video and Image Compression Coding

37. Simulation (3/3) • Overlapped Block Motion Compensation Deblocking Algorithms in Video and Image Compression Coding

38. Conclusions and Future Work • Many conventional methods do not exploit the characteristics of the blocking artifacts • The new algorithms apply different filter with different strength to different areas • Adopting the blockiness metric to measure the blockiness will be the future trend Deblocking Algorithms in Video and Image Compression Coding

39. References (1/2) In-loop Filter • [1] Lain E.G. Richardson, “H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia”, John Wiley & Sons, Jan. 2004. • [2] Lee, Jae-Beom, Kalva and Hari, ”The VC-1 and H.264 Video Compression Standards for Broadband Video Services”, Springer, 2008. • [3] “Text of ISO/IEC FDIS 14496-10/Draft ITU-T H.264: Information Technology – Coding of Audio-Visual Objects: Advanced Video Coding”, International Organization for Standardization, 2003. • [4] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhanUrhan, SarpErtürk and Tae-Gyu Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007. • [5] S. Romero, and L.F. Romero, “An Optimized Preconditioned Conjugate Gradient Algorithm”, Technicah Report No: UMA-DAC-02/11, University of Malaga, Sept. 2002. Post-Filtering • [6] Tao Chen, Hong Ren Wu and Bin Qiu, “Adaptive Postfiltering of Transform Coefficients for the Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 11, no. 5, Dec. 2001. • [7] Shizhong Liu and Alan C. Bovik, “Efficient DCT-Domain Blind Measurement and Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 12, no. 12, May 2002. • [8] Ci Wang, Wen-Jun Zhang and Xiang-Zhong Fang, “Adaptive Reduction of Blocking Artifacts in DCT Domain for Highly Compressed Images”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 50, no. 2, May 2004. • [9] A. Z. Averbuch, A. Schclar and D. L. Donoho, “Deblocking of block-transform compressed images using weighted sums of symmetrically aligned pixels,” IEEE Trans. on Circuits Syst. Video Technology, vol.14, pp.200-212, Feb. 2005. Deblocking Algorithms in Video and Image Compression Coding

40. References (2/2) • [10] Jongho Kim, MinseokChoi, and JechangJeong, “Reduction of Blocking Artifact for HDTV using Offset-and-Shift Techniques”, IEEE Trans. on Consumer Electronics, vol. 53, no.4, November 2007. • [11] ZixiangXiong, Michael T. Orchard, and Ya-Qin Zhang, “A Deblocking Algorithm for JPEG-Compressed Images Using Overcomplete Wavelet Representations”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 7, no. 2, April 1997. • [12] Jongho Kim and JechangJeong, “Adaptive Deblocking Technique for Mobile Video”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007. Pre-Filtering • [13] Sheng-Ho Wang, Sung-Wen Wang, Yi-Shin Tung, Ja-Ling Wu, Jau-Hsiung Huang, "Pre-Process for maximizing the effect of in-loop deblocking filtering in H.264/AVC encoding," 5th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services (EC-SIP-M 2005), Smolenice, Slovak, 29 June - 2 July, 2005. Overlapped Block Methods • [14] H. S. Malvar and D. H. Staelin, “The LOT: transform coding without blocking effects”, IEEE Trans. on Acoustic., Speech, Signal Processing, vol. 37, no. 4, pp. 553-559, April 1989. • [15] M. T. Orchard and G. J. Sullivan, “Overlapped block motion compensation: an estimation-theoretic approach”, IEEE Trans. on Image Processing, vol. 3, no. 4, pp. 693-699, Sep. 1994. Block-Edge Impairment Metric • [16] H. R. Wu and M. Yuen, “A Generalized Block-Edge Impairment Metric for Video Coding”, IEEE Signal Processing Letters, vol. 4, no. 11, Nov. 1997. • [17] AthanasiosLeontaris, Pamela C. Cosman and Amy R. Reibman, “Quality Evaluation of Motion-Compensated Edge Artifacts in Compressed Video”, IEEE Trans. on Image Processing, vol. 16, no. 4, April 2007. Deblocking Algorithms in Video and Image Compression Coding