Unified Loop Filter for High-performance Video Coding

1. Unified Loop Filter for High-performance Video Coding Yu Liu and Yan Huo ICME2010, July 19-23, Singapore

2. Outline • Introduction • Proposed Unified Loop Filter • Experimental Results • Conclusion

3. Introduction • Conventional Video Coding Standard • Block-based DPCM coding • Transform, quantization, ME/MC, in-loop deblocking filter, entropy coding • In-loop Deblocking Filter • A bank of fixed low-pass filters to alleviate blocking artifacts • Assume smooth image model • thus singularities, such as edges and textures, are not handled correctly • an analysis on the gradients across the boundary is performed to check whether the filtering should be skipped to preserve image sharpness. • Pre-defined filter coefficients • do not retain the frequency-selective properties or have the ability to suppress quantization noise optimally

4. Introduction • Adaptive Wiener Filter [4-7] • Well-known optimal linear filter: adaptive post/loop filter • Improve the quality of reconstructed picture degraded by compression • Pros: • Guarantee the optimized objective quality restoration • Cons: • Can’t efficiently improve the subjective quality if used alone; thus • Have to be utilized on top of in-loop deblocking filtered picture to achieve both improved objective and subjective quality

5. Introduction • Proposed Unified Loop Filter • Is there a way to combine the advantages of deblocking filter and Wiener filter into a unified filtering framework? • Motivation • The fact: multiple sources of information loss in current video coding standards, such as H.264/AVC • Quantization: the only source of information loss, prior to H.264/AVC • Deblocking loop filter in H.264/AVC: another source of information loss • Adaptive loop filter in KTA: • may not be able to reach the capability upper-bound of picture restoration • due to additional information loss brought by the deblocking filtering • Unified Loop Filter • Reduces the number of the sources causing information loss • Thus further improves the capability of picture restoration

6. Introduction • Block Diagram of Conventional Video Codec

7. Introduction • Block Diagram of Video Codec with Unified Loop Filter

8. Proposed Unified Loop Filter • Order Statistics Filter • Filters utilizing order statistics information [9], improved on median filters, can effectively remove the blocking artifacts and ringing artifacts, while retaining the sharpness of edges. • Although order statistics filter is a nonlinear filter, it can be optimized to minimize the mean square error by using linear combination of ordered statistics. • Here, the order statistics filter is used to combine nonlinear enhancement filter and linear restoration filter into one unified filtering framework. • Unified Loop Filter • Suppose that X=(x1, x2, …,xc,… , xN)T is a support vector containing N pixels of the reconstructed picture arranged by the spatial order surrounding the central pixel xc. • The unified loop filter is constructed as follows:

9. Proposed Unified Loop Filter • Step 1: Vector of Similarity Statistics • The support vector X is converted to form a vector of similarity statistics X’=(x’1, x’2, … , x’N)T by using the following equation: where f(xc,xi) is the similarity function. Real-valued similarity functions have to satisfy the following constrains: • In this paper, the following similarity function is adopted: where σ is the spread parameter controlling the strength of similarity function.

10. Proposed Unified Loop Filter • Step 2: Similarity-Ordered Statistics Filter • The vector of similarity statistics X’ is further ordered to form a vector of similarity-ordered statistics Xn=(x’(1),x’(2), …, x’(N))T by using the following rule: • Then the output of non-linear similarity-ordered statistics filter becomes where Wn is the vector of N optimized filter coefficients.

11. Nonlinear Part Linear Part Proposed Unified Loop Filter • Step 3: Unified Loop Filter • In order to improve the coding efficiency, linear Wiener filter should also be incorporated into the unified loop filter. Suppose that Xl=(x1, x2, …,xc,… , xM)T is the support vector of Wiener filter, the output of Wiener filter becomes • where Wl is a vector of M optimized filter coefficients. • Generally speaking, Wiener filter is also a kind of order statistics filter, called as spatially ordered statistics filter, because its support vector Xl is constructed by arranging the M pixels in a spatial order. Therefore, nonlinear similarity-ordered statistics filter is concatenated with linear spatially ordered statistics filter, aka Wiener filter, to form the unified loop filter: • where Xu =(x’(1), x’(2), … , x’(N) , x1, x2, … , xM )T and Wu is a vector of M+N optimized filter coefficients.

12. Proposed Unified Loop Filter • Optimization of Unified Loop Filter • The optimization of the unified loop filter falls into the classical optimization framework of least mean square error (LMSE). The solution can be obtained by solving the Wiener-Hopf equations: • where xo is the original video frame. The minimization problem can be solved by the Wiener-Hopf equation, which is given by • where Ru,u(k,l) is the auto-correlation function of xu, which is defined as • and Ro,u(l) is the cross-correlation function between xo and xu , which is defined as

13. Proposed Unified Loop Filter • Filter Design • Two considerations in the filter design of unified loop filter: • not only the subjective enhancement (for removing the blocking and ringing artifacts) • but also the objective restoration (for improving the coding efficiency) • For Luma Component (Y) • In unified loop filter, nonlinear part consists of one 12-tap diamond filter, and linear part consists of four kinds of different taps (1-tap, 13-tap, 25-tap, and 41-tap) diamond filters with central point symmetry or or or Linear part Nonlinear part

14. Proposed Unified Loop Filter • Filter Design • For Chroma Components (Cr/Cb) • In unified loop filter, nonlinear part consists of one 4-tap diamond filter, and linear part consists of two kinds of different taps (1-tap and 13-tap) diamond filters with central point symmetry • Selection of Filter Tap Type • The tap type of linear part in unified loop filter is decided by rate-distortion optimization selection within the whole frame: • The filter side information includes the filter tap type and filter coefficient quantization bits, which are encoded and transmitted to the decoder side. or Nonlinear part Linear part

15. Experimental Results • Test Conditions • The proposed unified loop filter has been implemented within JM11.0KTA2.4r1 reference software. The test conditions are listed as follows: Table 1. Test conditions

16. Experimental Results Table 2. Coding gain comparison of different coding schemes, compared with H.264/AVC High Profile, in BD bitrate reduction (BR) and BD-PSNR gain (for Luma) • Objective Performance Comparison

17. Experimental Results • Subjective Quality Comparison (a) Anchor (b) ALF w/o DLF (c) ALF + DLF (e) ULF Figure. Part (128x128) of the reconstructed SpinCalendar sequence at the 54th frame with QP=38.

18. Experimental Results • Complexity Comparison • Compared with “ALF+DLF”, extra complexity introduced by ULF includes: • Similarity computation: lookup-table (LUT) technique • Sorting process: counting sort algorithm with linear complexity O(n). • With the help of the two fast algorithms • ULF only increases the execution time 13.70% and 26.52% on average for the encoder and the decoder, respectively, compared with “ALF+DLF” Table 3. Average execution time comparison of different coding schemes

19. Conclusion • Unified Loop Filter (ULF) • Combine the advantages of linear filter and nonlinear filter to achieve both objective and subjective quality optimization • The deblocking loop filter in conventional video codec can be removed, and thus replaced by the proposed unified loop filter • The proposed unified loop filter can be used in any hybrid video coding system • Classification-based Unified Loop Filter (CULF) • Global v.s. Local: different regions have different quantization error characteristics • It is better to classify pixels into different groups: • one group for boundary pixels with blocking artifacts • one group for non-boundary and boundary pixels without blocking artifacts • other group for non-filtering pixels • Different unified loop filter with different characteristics is applied to each group • Additional 1.57% bitrate reduction is achieved, compared with ULF