Scalable Extension of H.264/AVC

1. Scalable Extension of H.264/AVC Student: Hung-Chih Lin Advisor: Prof. Hsueh-Ming Hang

2. References • [1] Reichel, J., Hanke, K., Popescu, B.: Scalable Video Coding V1.0. ISO/IEC JTC1/SC29/WG11, N6372 (2004) • [2] H. Schwarz, D. Marpe, and T. Wiegand, “Scalable Extension of H.264/AVC”, ISO/IEC JTC1/WG11 Doc. M10569/S03, Mar. 2004. • [3] I. Daubechies and W. Sweldens, “Factoring wavelet transforms into lifting steps”, J. Fourier Anal. Appl. 4(3), pp. 245-267, 1998. • [4] J. Reichel, H. Schwarz, and M.Wien, "Joint Scalable Video Model JSVM-2," 17th JVT meeting, JVT-Q202, Nice, France. • [5] Tabatabai, A., Visharam, Z., Suzuki, T.: Compariosn of MCTF and closed-loop hierarchical B pictures. ISO/IEC JTC/SC29/WG11 and ITU-T SG16 Q.6, JVT-P059 (2005) hclin

3. Outline • Overview • MCTF in JSVM • Scalability Concepts • JSVM Reference Software hclin

4. Outline • Overview • Motivation • Scalable Video Coding • MCTF in JSVM • Scalability Concepts • JSVM Reference Software hclin

5. Motivation • To support clients with diverse capabilities in complexity, bandwidth, power, and display resolution. hclin

6. Scalable Video Coding • Approaches • wavelet-based • 2D+t structure • t+2D structure • AVC-based • Layered coding concept hclin

7. Approaches A wavelet-based approach with 2D+t structure hclin

8. Approaches A wavelet-based approach with t+2D structure hclin

9. Approaches An AVC/H.264-based structure hclin

10. Scalabilities • Temporal • fps • Spatial • resolution • SNR/Rate • quality hclin

11. Outline • Overview • MCTF in JSVM • Why MCTF ? • Base layer structure • Inter layer prediction • Adaptive Prediction/Update Steps • Progressive MCTF • Scalability Concepts • JSVM Reference Software hclin

12. Why MCTF? • MCTF = Motion-Compensated Temporal Filtering • A temporal sub-band coding • 2-channel filter bank in temporal direction • Performs the wavelet decomposition / reconstruction along the motion trajectory • Implementation technique • Lifting scheme(the main reason) : Any bi-orthogonal wavelet filters can be factorized by prediction and update steps hclin

13. Lifting scheme • Attraction • An in-place implementation like FFT. • Easy to build non-linear WT. • Insure PR. • All operations within one lifting step can be done entirely parallel. • Computational complexity • ~40% of original one (depend on the wavelet filter) hclin

14. Lifting scheme hclin

15. Factoring Wavelet Transforms into Lifting Steps • 2-channel Filter Bank • Bi-orthogonal x[n] H0(z) 2 2 F0(z) y[n] H1(z) 2 2 F1(z) x[n] 2 2 y[n] 2 2 hclin

16. Factoring Wavelet Transforms into Lifting Steps • PR condition • Define hclin

17. Factoring Wavelet Transforms into Lifting Steps • Type 1 polyphase representation • Define hclin

18. Factoring Wavelet Transforms into Lifting Steps • Noble identities L H(zL) H(z) L H(zM) M M H(z) hclin

19. Factoring Wavelet Transforms into Lifting Steps • We want and are FIR. • By Euclidean algorithm, we can get hclin

20. Factoring Wavelet Transforms into Lifting Steps hclin

21. Wavelet filters • 2-2 Filter Bank (Haar) • 5-3 Filter Bank hclin

22. Lifting scheme hclin

23. MCTF 30Hz Video Sequence 15Hz Video Sequence H H H H H H H H1 7.5Hz Video Sequence H2 H2 H2 H2 3.25Hz Video Sequence H3 H3 H4 L hclin

24. MCTF (a) Without M.C. (b) With M.C. hclin

25. Lifting scheme hclin

26. Base layer Structure • Compatible with AVC Main profile • Dyadic hierarchical B pictures • Only prediction step is performed. (UMCTF) hclin

27. Base layer Structure • Non-dyadic decomposition is available • Temporal scalability hclin

28. Inter Layer Prediction • Remove the redundancy among the different layers • Residues • Motion vectors hclin

29. Inter Layer Prediction hclin

30. Adaptive Prediction/Update Steps • Goal • Control the encoding delay • Method • GOP is partitioned into sub-groups • Restrictions : no across the partition boundary • Backward prediction steps • Backward and forward update steps hclin

31. Adaptive Prediction/Update Steps hclin

32. Progressive MCTF • Prediction steps and update steps are interlaced. • Process the pictures in the reverse display order. hclin

33. Progressive MCTF hclin

34. Outline • Overview • OMCTF in JSVM • Scalability Concepts • Three Scalabilities • Slice Types • Combined scalability • JSVM Reference Software hclin

35. Temporal Scalability hclin

36. Spatial Scalability Interpolation filter: {1,-5,20,20,-5,1} hclin

37. SNR Scalability hclin

38. Slice Types hclin

39. Slice Types hclin

40. Combined Scalability hclin

41. Outline • Overview • OMCTF in JSVM • Scalability Concepts • JSVM Reference Software • Tools • UMCTF at Decoder hclin

42. Tools • Converter • Spatial domain • Upsample • Interpolation FIR filter • Downsample • Apply an anti-aliasing FIR filter proir to 2D downsampling • Temporal domain • PSNR hclin

43. UMCTF at Decoder • Update step • Improve coding efficiency • Increase significantly complexity of the decoder operation • Additional M.C. operations • Picture buffer management • M.V. needs intensive branch instructions hclin

44. UMCTF at Decoder • UMCTF => update step at decoder side is omitted • The visual quality and PSNR of the decoded video is not degraded • UMCTF → purely predictive structure • Reduce the complexity of decoder by 50% hclin

45. Normal Mode hclin

46. High Quality (Qp = 0) hclin

47. Qp = 24 hclin

48. Thank you !! hclin