Transcoding from H.264/AVC to HEVC

1. Transcoding from H.264/AVC to HEVC Shantanu Kulkarni UTA ID: 1000789943

2. Introduction to Transcoding • Video transcoding is the operation of converting video from one format to another. • A format is defined by characteristics such as bit-rate, spatial resolution etc. • Transcoding is one of the most promising technologies, which provides video adaptation in terms of bit-rate reduction, resolution reduction and format conversion to meet various requirements

3. Following is the most basic transcoding architecture Fig. 1 Basic architecture for transcoding [8]

4. Need for Transcoding • Design of most video coding standards is primarily aimed at having the highest coding efficiency, which is the ability to encode the video at lowest possible bitrate while maintaining certain level of video quality. • HEVC, which is a recently emerged video coding standard, aims at high coding efficiency while retaining the video quality • With its hybrid coding architecture, motion compensation prediction and transform coding technique, it can be seen as an improved version of the previous standard H.264 [6]

5. Need for Transcoding contd. • Transcoding from H.264 to HEVC will enable lowering the bitrate resulting in a more efficient compression.[1] • AVC and HEVC share a similar prediction, transform, quantization, and entropy coding architecture. [1]

6. Overview of HEVC • The HEVC standard is based on the well-known block-based hybrid coding architecture, combining motion-compensated prediction and transform coding with high-efficiency entropy coding • It employs a flexible quad-tree coding block partitioning structure that enables the efficient use of large and multiple sizes of coding, prediction, and transform blocks. • It also employsimproved intra prediction and coding, adaptive motion parameter prediction and coding, new loop filter and an enhanced version of context-adaptive binary arithmetic coding (CABAC) entropy coding.

7. Overview of HEVC contd. Fig. 2 HEVC encoder block diagram

8. Overview of HEVC contd. Fig. 3 Block based architecture HEVC – Intra-Prediction [17]

9. Overview of H.264/AVC [6] • Directional spatial prediction for intra coding (9 directional prediction modes) • Variable block-size motion compensation with small block size • Quarter-sample-accurate motion compensation • Motion vectors over picture boundaries • Multiple reference picture motion compensation • Decoupling of referencing order from display order • In-the-loop deblocking filtering

10. H.264/AVC Encoder-Decoder Block Diagram Fig. 4 H.264/AVC Encoder [2] Fig. 5 H.264/AVC Decoder [2]

11. Comparison of AVC & HEVC • Larger block structure leading to maximum of 64x64 pixels per block • Intra prediction direction modes which are upto 35 (33 modes + DC + Planar) in case of HEVC while H.264 has 9 directional modes of intra prediction • Adaptive motion vector prediction, which allows codec to find more inter frame redundancies • Superior parallelization tools, including wavefront parallel processing, for more efficient coding in a multi core environment • Entropy using CABAC only, no more CAVLC • Improvements to de-blocking filter and addition of one more filter called Sample Adaptive Offset (SAO) that further leaves artifacts along block edges

12. Output HEVC Bitstream Input AVC Bitstream AVC Decoder HEVC Re-encoder Residual, modes and MVs CU, PU partitions and MVs Simplified Mode Selection HEVC Transcoder • The transcoding schemes discussed here avoid high computational complexity in terms of reduced RDO evaluations and motion compensation operation as well as fractional pixel interpolation operation. Fig. 6 Pixel domain AVC-HEVC transcoder[1]

13. HEVC Transcoder Contd. • The LCU will initially split according to the input MB modes in AVC • The initial CU partitions will be further merged to larger size according to the predict directions of its adjacent four sub-CUs. • For example, if the predict directions of adjacent four 8x8 CUs are the same, they will be merged to 16x16. Similar merge operations will also perform on CUs larger than 8x8. The merge process is applied from the 4x4 smallest 4x4 blocks to the blocks with size 32x32. • The input information from AVC can be used to reduce the candidate predict directions for SATD, or reducing the candidate SATD list

14. HEVC Transcoder contd. • The major complexity of Inter picture coding comes from the motion estimation (ME), MC operations when testing every set of possible coding parameters with possible CU size, PU and TU modes. • Thus, it is proposed to reduce these operations with the help of input • AVC information, e.g. residuals, modes and MVs.Since the largest CU (LCU) in HEVC consists of 16 MBs in AVC, after AVC decoding, the information of these MBs will transmit to the mode selection module.

15. HEVC Transcoder Contd. • Since the input AVC bitstream already contains useful information of the MB partitions and prediction directions, we extract the information which can be utilized by HEVC encoder without having to do any computations. • Key technology of AVC to HEVC transcoding consists of merging smaller blocks to a larger CU, especially for bit rate reduction transcoding. Since a large CU may consists of different 4x4 blocks, and probably, these blocks may have different MVs, merging these blocks now turns to measure the RD cost when the MV changes.

16. Cascaded encoder decoder transcoder architecture • Includes complete decoding and re-encoding • High Complexity • Error due to the lossy encoding of already decoded bit sequence Cascaded Decoder and Encoder Transcoded HEVC Bit stream Input Bit stream H.264 Bit stream Output Bit stream H.264 Encoder H.264 Decoder HEVC Encoder HEVC Decoder Reconstructed Bit stream Fig.7 Cascaded encoder – decoder transcoder

17. Simulation results Table 1. MSE and PSNR of akiyo_qcif.yuv video sequence for 100 frames

18. Simulation results continued… Table 2. MSE and PSNR of foreman_qcif.yuv video sequence for 100 frames

19. Simulation results continued… Table 3. MSE and PSNR of mobile_cif.yuv video sequence for 100 frames

20. Simulation results continued… Table 4. MSE and PSNR of coastguard_cif.yuv video sequence for 100 frames

21. Comparison of PSNR for colored images between ‘qcif’ video sequences akiyo and foreman PSNR (db)  PSNR (dB) Video Sequences  Video Sequences  Fig 8. PSNR Video sequences: akiyo_qcif and foreman_qcif

22. Comparison of PSNR for colored images between ‘cif’ video sequences mobile and coastguard PSNR (dB) Video Sequences  Fig 9. PSNR Video sequences: mobile_cif and coastguard_cif

23. Comparison of bitrate for akiyo_qcif video sequence Bitrate (kbps) Video Sequences  Fig 10. Bitrate comparison between H.264 encoded, HEVC encoded and transcoded output using foreman_qcif.yuv sequence. (100 frames)

24. Comparison of bitrate for foreman_qcif video sequence Bitrate (kbps) Video Sequences  Fig 10. Bitrate comparison between H.264 encoded, HEVC encoded and transcoded output using foreman_qcif.yuv sequence. (100 frames)

25. Comparison of bitrate for mobile_cif video sequence Bitrate (kbps) Video Sequences  Fig 11. Bitrate comparison between H.264 encoded, HEVC encoded and transcoded output using mobile_cif.yuv sequence. (100 frames)

26. Comparison of bitrate for coastguard_cif video sequence Bitrate (kbps) Video Sequences  Fig 12. Bitrate comparison between H.264 encoded, HEVC encoded and transcoded output using coastguard_cif.yuv sequence. (100 frames)

27. Comparison of computation time for akiyo_qcif video sequence Time (sec) Video Sequences  Fig 13. Computation time comparison between H.264 encoded, HEVC encoded and transcoded output using akiyo_qcif.yuv sequence. (100 frames)

28. Comparison of computation time for foreman_qcif video sequence Time (sec) Video Sequences  Fig 14. Computation time comparison between H.264 encoded, HEVC encoded and transcoded output using foreman_qcif.yuv sequence. (100 frames)

29. Comparison of computation time for mobile_cif video sequence Time (sec) Video Sequences  Fig 15. Computation time comparison between H.264 encoded, HEVC encoded and transcoded output using mobile_cif.yuv sequence. (100 frames)

30. Comparison of computation time for coastguard_cif video sequence Time (sec) Video Sequences  Fig 16. Computation time comparison between H.264 encoded, HEVC encoded and transcoded output using coastguard_cif.yuv sequence. (100 frames)

31. SSIM Index of video sequences before and after open loop transcoding Video Sequences  Fig. 17 SSIM comparison between the H.264 encoded, HEVC encoded and transcoded frames

32. Images Akiyo_qcif and Foreman_qcif Fig 18a. Fig 18b. Fig 18c. Fig 18. Akiyo_qcif video sequence 18a) H.264 encoded, 18b) transcoded and 18c) HEVC encoded and reconstructed Fig 19a. Fig 19b. Fig 19c. Fig 19. Foreman_qcif video sequence 19a) H.264 encoded, 19b) transcoded and 19c) HEVC encoded and reconstructed

33. Images Mobile_cif Fig 20a. Mobile_cif H.264 encoded and reconstructed Fig 20b. Mobile_cif Transcoded Fig 20c. HEVC Encoded and reconstructed

34. Images coastguard_cif Fig 21a. Coastguard_cif H.264 encoded and reconstructed Fig 21b. Coastguard_cif Transcoded Fig 21c. Coastguard_cif HEVC Encoded and reconstructed

35. Proposed Transcoder Architecture and features • Heterogeneous Video Transcoder Architecture • Motion Vector reuse Technique • MV refinement using minimum SAD

36. Proposed Transcoder contd… Transcoded HEVC Bit stream Input Bit stream Output Bit stream H.264 Bit stream H.264 Encoder H.264 Decoder HEVC Encoder HEVC Decoder Store the MVs in a text file MV Refinement Reuse Motion vectors Store the MVs in a text file Fig. 22 Block diagram of the proposed transcoder with motion vector reuse

37. Heterogenous Video Transcoder Architecture [13] • ‘Heterogenous’ since it performs the conversion of bitstreams across the formats (like in this case, H.264 to HEVC) • It also provides the functionalities of homogenous transcoding , like bitrate reduction and spatio-temporal resolution reduction • The biggest difference between homogenous transcoding and heterogeneous transcoding is the presence of syntax conversion module in the heterogeneous model. • Heterogenous transcoding is achieved through complete decoder from the source stream and reduced encoder for the transcoded stream, that reuses information present in the source bitstream to speed up the transcoding. • Decoder extracts some information from the received bitstream like motion vectors and mode information for the macro blocks and these are processed according the target transcoder format. This include reduction in complexity as compared to the cascaded encoder-decoder transcoder.

38. Heterogenous Video Transcoder Architecture contd… Fig 23 Block diagram of heterogeneous video transcoder [ 13]

39. Motion Vector Reuse • The maximum encoding time in HEVC encoder is consumed by the motion vector computation process. • Aim of this technique - to import the motion vectors from the H.264 decoder and reuse them into HEVC encoder. • As a result, the motion vector computation time is saved, complexity of the encoder is reduced and overall computation time is reduced.

40. Motion vector reuse contd… Two parts:- • Extracting the motion vector from H.264 • Mapping the macroblocks, MB modes and motion vectors from H.264 to HEVC.

41. Motion Vector Reuse contd… Extracting the motion vectors from the H.264 • Following items are stored in the text file -Slice number -MB number -MB type -SMB type -Reference Picture Index -Motion Vector x,y

42. Fig 24 Screen shot of the extracted macroblock and MV information from H.264 decoder (Intra - MB)

43. Fig 25 Screen shot of the extracted macroblock and MV information from H.264 decoder (Inter - MB)

44. Motion Vector Reuse contd… MB mapping from H.264 to HEVC - To account for the spatial resolution reduction 16 16 (a) 64 64 (b) Fig 26 (a) MB size in H.264 and (b) LCU size in HEVC

45. Motion Vector Reuse contd… • For every MB, one motion vector is calculated. • If the MB has a 16x8 or 8x16 or 8x8 partition, then motion vector for the MB is the average of all the motion vectors in the MB. (a) 16x16 MB (b) 16x8 MB (c) 8x8 MB (d) 8x16 MB Fig 27 the MB type for H.264 inter prediction For (b), H.264mv_x = (H.264mv_ax+H.264mv_bx)/2 H.264mv_y = (H.264mv_ay+H.264mv_by)/2 For (c), H.264mv_x = (H.264mv_ax+H.264mv_bx)/4 H.264mv_y = (H.264mv_ay+H.264mv_by)/4 For (d), H.264mv_x = (H.264mv_ax+H.264mv_bx)/2 H.264mv_y = (H.264mv_ay+H.264mv_by)/2

46. Motion Vector reuse contd… Mapping of H.264 MB to HEVC LCUs Assumption: The AMP (Asymmetric Motion Prediction) mode is disabled. AB Average of all 16 MBs of H.264 HEVCmv_A = Avg of mvs of 8 MB HEVCmv_B = Avg of mvs of 8 MB (a) 64x64 LCU (b) 32x64 CU A B HEVCmv_A = HEVCmv_B= HEVCmv_C = HEVCmv_D = Avg of mvs of 8 MB HEVCmv_A = Avg of mvs of 8 MB A B C D HEVCmv_B = Avg of mvs of 8 MB (c) 64x32 CU (d) 32x32 CU Fig 28 Partitioning of HEVC CU and the motion vectoring mapping

47. Motion vector reuse contd… • Similarly for the asymmetric partitioning modes of CUs. Fig 29 Asymmetric partitioning of the motion vectors

48. Motion Vector Reuse contd… • It is possible that the motion vector cannot directly be reused because the original reference frame is dropped, which can be a possibility in case of temporal reduction while encoding. • In such cases, motion vectors can be exploited to derive new motion vectors to be used in the transcoder. -- Motion Vector Scaling

49. Motion vector reuse contd… Motion Vector Scaling Technique --The movement between the frames is modeled linear and the MV is just scaled to the new reference frame. --Low complexity --Always produces candidate MV, given a starting MV Fig 30 Motion vector scaling algorithm diagram The scale factor is always proportinal to the distance between the original and target reference frames and the current frames Where, n = current frame n-a = original reference frame n-ß = target reference frame

50. Motion vector refinement [14], [15] • There is a possibility that the motion vectors derived by motion vector reuse are non-optimal, due to quantization errors. • To obtain the optimal motion vector, motion vector computation can be done using a small search range rather than a full search algorithm. • Use of minimum SAD algorithm to refine the motion vector and search range of -2 and +2 pixels around the incoming motion vectors can be used [14].