1 / 29

EE 5359-MULTIMEDIA PROCESSING

3D EXTENSION of HEVC: Multi-View plus Depth. EE 5359-MULTIMEDIA PROCESSING. Parashar Nayana Karunakar Student Id: 1000833406 Department of Electrical Engineering. HEVC & MVD- Brief overview.

jun
Download Presentation

EE 5359-MULTIMEDIA PROCESSING

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3D EXTENSION of HEVC: Multi-View plus Depth EE 5359-MULTIMEDIA PROCESSING ParasharNayanaKarunakar Student Id: 1000833406 Department of Electrical Engineering

  2. HEVC & MVD- Brief overview • High-Efficiency Video Coding (HEVC) is the newest video coding standard of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). • The Joint Collaborative Team on 3D Video Coding Extension • Development was created to develop 3D video coding technology • more advanced than the current multiview video coding (MVC) features • of H.264. The standards for which these 3D video coding extension • technologies will provide such enhanced capabilities may include H.262, • H.264 and the High Efficiency Video Coding (HEVC) .

  3. HEVC – ENCODER Block DIAGRAM Encoder block diagram H.264 [18] Fig 1. – Typical HEVC Encoder[1]

  4. What is 3D/Multi-View/Stereo Video • Multiple Cameras views of the same scene are captured – Multiple View Video(MVV) • Efficient Compression techniques are essential as MVV contains vast amount of data • both during storage and transmission. • Inter-view statistical dependencies are exploited for combined temporal/inter-view • prediction. • When color video and an associated per sample depth map are considered, we get • Multi-view video plus depth representation. Fig 2. – Test Sequence –Balloons with depth map [3]

  5. Fig 3. – Overview of the system structure and data format for the transmission of 3D video [12]

  6. MVC for MULTI-VIEW PLUS DEPTH • Encoding and decoding of each view of a multi-view data set separately ,referred to as simulcast coding , can be done with any video codec including H.264/AVC and HEVC. • This would be simple but inefficient as inter-view statistical dependencies are not exploited. • In order to exploit all the statistical dependencies within multi-view dataset, inter-view prediction has to be combined with temporal prediction. • As seen in fig 3b. , In MVC, one of the views is conventionally coded in conformance to the HEVC codec. For coding the other views, in addition to previously coded pictures of the same view already coded co-located pictures of other views can also be used as reference pictures. • In Multi –view video plus depth (MVD) format, only a few views are actually coded. Based on the transmitted videos and depth maps, additional views can be rendered.. Fig 4b. – Multi-view coding structure with hierarchical B pictures for both temporal (black arrows) and inter-view prediction(red arrows) [2] Fig 4a. – Simulcast coding structure with hierarchical B pictures for temporal prediction(black arrows)[2]

  7. Basic 3d video codec structure Fig 5. – Block Diagram of a 3D Video Codec[4]

  8. Mvd codec- working • The basic structure of the 3D video codec is shown in the block diagram of Figure 5. In • principle, each component signal is coded using an HEVC-based codec. The resulting • bit stream packets, or more accurately, the resulting Network Abstraction Layer (NAL) • units, are multiplexed to form the 3D video bit stream. • The base or independent view is coded using an unmodified HEVC codec. The base view • sub-stream can be directly decoded using the conventional HEVC decoder. • For coding the dependent views and the depth data, modified HEVC codec are used, • which are extended by including additional coding tools and inter-component prediction • techniques that employ already coded data inside the same access unit as indicated by • the red arrows in Figure 5. • For enabling an optional discarding of depth data from the bit stream, e.g., for • supporting the decoding of a stereo video suitable for conventional stereo displays, • the inter-component prediction can be configured in a way that video pictures can • be decoded independently of the depth data..

  9. MVD- CODING ALGORITHM • The video pictures and, when present, the depth maps are coded access unit by access unit, as it is illustrated • in Figure 6. • An access unit includes all video pictures and depth maps that correspond to the same time instant. NAL • units containing camera parameters may be additionally associated with an access unit. • The video pictures and depth maps corresponding to a particular camera position are indicated by a view • identifier (viewId). All video pictures and depth maps that belong to the same camera position are associated • with the same value of viewId. • Inside an access unit, the video picture and, when present, the associated depth map with viewId equal to 0 are • coded first, followed by the video picture and depth map with viewId equal to 1, etc. • For ordering the reconstructed video pictures and depth map after decoding, each value of viewId is associated • with another identifier called view order index (VOI). The view order index is a signed integer values, which specifies • the ordering of the coded views from left to right. Fig 6. - Access units structure and coding order of view components[12]

  10. COMPARSION – MVD AND HEVC CODEC • CODING OF DEPENDENT VIEWS -- Additional tools have been integrated into the HEVC • codec, which employ already coded data in other views for efficiently representing a • dependent view. These tools include - Disparity-compensated prediction, • View synthesis based inter-view prediction, Post processing in-loop filtering, • Inter-view motion prediction, Depth-based motion parameter prediction, • Inter-view residual prediction, Adjustment of QP of texture based on depth data. • CODING OF DEPTH MAPS – There are certain additional tools and also some tools are • removed for coding of Depth maps. Some of the differences are -- Depth Maps are • coded in 4:0:0 format, Non-linear depth representation is used, Z-near Z-far • compensated weighted prediction, Modified motion compensation and motion vector • coding ( No interpolation is used i.e. for depth maps, the inter-picture prediction is • always performed with full-sample accuracy. Disabling of in-loop filtering ( deblocking • filter and SAO), Depth modeling modes ( Four new Intra-prediction modes are used), • Motion parameter inheritance.

  11. THE PROPOSAL • The aim of the project is to import the MVD coding tools into the HM 9.1. • The project will be carried out in the following steps: • Learn about the tools that are different to MVD when compared to • standard HEVC codec. • Study and compare the MVD extensions of HEVC carried out on HM 5.1 • by Fraunhoffer HHI and Qualcomm ( the two most recent ones). Import the MVD to HM 9.1 based on the observations from the first two • stages. • Present the results and report the changes in bitrates and PSNR between the • new imported software and the previously studied

  12. RESULTS • The sequence balloons (1024 x 768) has been used. The tested configuration is the • balloons sequence with three camera view including depth with QP of 30. The same • configuration has been used with the sequence kendo (1024 x 768 ) as well. • The encoding is done using both HTM 5.1 and HTM 6.0 for comparison. Finally, the • encoding is done using the modified code which will be referred to as “mychanges” • henceforth. • The results are as shown in the next few slides,

  13. RESULTS – Encoding Time (sec) Comparison for Balloons Sequence S E C O N D S Fig 7. Encoding Time in Seconds for Balloons Sequence

  14. RESULTS – Encoding Time (sec) Comparison for Kendo Sequence S E C O N D S Fig 8. Encoding Time in Seconds for Kendo Sequence

  15. RESULTS – Bit-rate (kbps) Comparison for Balloons Sequence Fig 9. Bit-Rate (kbps) for Balloons Sequence

  16. RESULTS – Bit-rate (kbps) Comparison for Kendo Sequence Fig 10. Bit-Rate (kbps) for Kendo Sequence

  17. RESULTS – Y-PSNR (dB) Comparison for Balloons Sequence Fig 11. Y-PSNR ( dB ) for Balloons Sequence

  18. RESULTS – Y-PSNR (dB) Comparison for Kendo Sequence Fig 12. Y-PSNR ( dB ) for Kendo Sequence

  19. Fig 13. -Video used – before and after compression – htm 5.1 Artifacts

  20. Fig 14. - Video used – before and after compression – htm 6.0 Artifacts Artifacts

  21. Fig 15. - Video used – before and after compression – MyChanges Artifacts

  22. Fig 16. - Video used – before and after compression – HTM 5.1-KENDO Artifacts

  23. Fig 17. - Video used – before and after compression – HTM 6.0 -KENDO

  24. Fig 18. - Video used – before and after compression – MYCHANGES -KENDO

  25. CODING STRUCTURE – HM 9.2 CODING STRUCTURE - MYCHANGES // coding structure Intm_iIntraPeriod; Intm_iDecodingRefreshType; Intm_iGOPSize; Intm_extraRPSs; GOPEntrym_GOPList[MAX_GOP]; Intm_numReorderPics[MAX_TLAYER]; Intm_maxDecPicBuffering[MAX_TLAYER]; Boolm_bUseLComb; Boolm_useTransformSkip; Boolm_useTransformSkipFast; Boolm_enableAMP; // coding structure Intm_iIntraPeriod; Intm_iDecodingRefreshType; Intm_iGOPSize; Intm_extraRPSs[MAX_VIEW_NUM]; GOPEntryMvcm_GOPListsMvc[MAX_VIEW_NUM][MAX_GOP+1]; Intm_numReorderPics[MAX_VIEW_NUM][MAX_TLAYER]; Intm_maxDecPicBuffering[MAX_VIEW_NUM][MAX_TLAYER]; Boolm_bUseLComb; Boolm_useTransformSkip; Boolm_useTransformSkipFast; Boolm_enableAMP; An example of difference in coding approach

  26. CONCLUSION • The goal of porting the 3D compression tools to HM 9.2 has been successfully • accomplished. • From the results, its clear that the changes made are not completely optimized and • hence the encoding time is the largest. • Large encoding time can be accounted for the fact that the project has achieved a • slightly better PSNR for depth maps compression compared to the previous versions used • for reference.

  27. FUTURE WORK • USE OTHER DIFFERENT SEQUENCES TO COMPARE AND UNDERSTAND THE RESULTS BETTER. • GENERATING OF THE ADDITIONAL VIEWS USING THE COMPRESSED VIEWS-RENDERING. • IMPROVE ENCODING TIME– MANY METHODS BEING USED IN HEVC COMPRESSION TECHNIQUES • CAN USED TO IMPROVE ENCODING TIME IN 3D COMPRESSION AS WELL. • USE OF PARALLELIZATION TECHNIQUES AND OTHER FAST ENCODING METHODS. • DEPTH MAPS CAN BE MADE MORE ARTIFACT FREE. • IN SHORT, 3D COMPRESSION IS A FERTILE FIELD FOR RESEARCH WITH A LOT OF SCOPE.

  28. AVC: Advanced video coding DIBR: Depth Image Base Rendering HD: High Definition HEVC: High Efficiency Video Coding HHI:Heinrich Hertz Institute HM: HEVC Test Model HTM: Test Model for 3D extension of HEVC IEC:  International Electrotechnical Commission ISO: International Organization for Standardization ITU-T: International Telecommunication Union-Telecommunication MPEG: Moving picture experts group MC: Motion Compensation MV: Motion Vector MVC: Multi-View Coding MVD: Multi-View plus Depth MVV: Multi-View Video NAL: Network Abstraction Layer PSNR: Peak Signal to Noise Ratio QP: Quantization Parameter SAO: Sample Adaptive Offset VCEG: Video coding experts group VOI: View Order Index List of acronyms

  29. [1] G.J. Sullivan; J. Ohm; Woo-Jin Han and T.Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Transactions on Circuits and Systems for Video Technology, Volume: 22, Issue: 12, Pages 1649-1668, December 2012. [2] P. Merkle, A Smolic, K. Müller, and T. Wiegand, “Multi-View video plus depth data representation and coding”. Picture Coding Symposium,2007 [3]Test Sequences: http://www.tanimoto.nuee.nagoya-u.ac.jp/~fukushima/mpegftv/ [4] H. Schwarz et al "3D Video Coding Using Advanced Prediction, Depth Modeling, and Encoder Control Methods", Picture Coding Symposium, May 2012. [5] G. Tech, H. Schwarz, K. Müller, and T. Wiegand, "Effects of synthesized View Distortion based 3D Video Coding on the Quality of interpolated and extrapolated Views", IEEE Intl. Conf. on Multimedia and Exposition, Pages 634-639, July 2012. [6] P. Merkle e tal, "3D Video: Depth Coding Based on Inter-component Prediction of Block Partitions", Picture Coding Symposium, May 2012. [7] H. Schwarz and T. Wiegand, "Inter-View Prediction of Motion Data in Multiview Video Coding", Picture Coding Symposium, May 2012. [8] G. Tech, H. Schwarz, K. Müller, and T. Wiegand, "3D Video Coding using the Synthesized View Distortion Change", Picture Coding Symposium, May 2012. [9] M. Winken, H. Schwarz, and T. Wiegand, "Motion Vector Inheritance for High Efficiency 3D Video plus Depth Coding," Picture Coding Symposium, May 2012. [10] S. Bosse, H. Schwarz, T. Hinz and T. Wiegand, "Encoder Control for Renderable Regions in High Efficiency Multiview Video Plus Depth Coding", Picture Coding Symposium, May 2012. [11] 3D Extension Software Repository: https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware/ [12] “Test Model under Consideration for HEVC based 3D video coding”, ISO/IEC JTC1/SC29/WG11 MPEG2011/N12559 February 2012, San Jose, CA, USA [13] HM Software Repository:https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ [14] HEVC Text Specification Draft 9: http://phenix.int-evry.fr/jct/doc_end_user/current_document.php?id=6803 [15] H.264/AVC reference website -http://www.itu.int/rec/T-REC-H.264-201003-I References

More Related