1 / 20

EE 5359-MULTIMEDIA PROCESSING

3D EXTENSION of HEVC: Multi-View plus Depth. EE 5359-MULTIMEDIA PROCESSING. Parashar Nayana Karunakar Student Id: 1000833406 Department of Electrical Engineering. HEVC & MVD- Brief overview.

michon
Download Presentation

EE 5359-MULTIMEDIA PROCESSING

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3D EXTENSION of HEVC: Multi-View plus Depth EE 5359-MULTIMEDIA PROCESSING ParasharNayanaKarunakar Student Id: 1000833406 Department of Electrical Engineering

  2. HEVC & MVD- Brief overview • High-Efficiency Video Coding (HEVC) is the newest video coding standard of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). • The Joint Collaborative Team on 3D Video Coding Extension • Development was created to develop 3D video coding technology • more advanced than the current multiview video coding (MVC) features • of H.264. The standards for which these 3D video coding extension • technologies will provide such enhanced capabilities may include H.262, • H.264 and the High Efficiency Video Coding (HEVC) .

  3. HEVC – ENCODER Block DIAGRAM Encoder block diagram H.264 [18] Fig 1. – Typical HEVC Encoder[1]

  4. What is 3D/Multi-View/Stereo Video • Multiple Cameras views of the same scene are captured – Multiple View Video(MVV) • Efficient Compression techniques are essential as MVV contains vast amount of data • both during storage and transmission. • Inter-view statistical dependencies are exploited for combined temporal/inter-view • prediction. • When color video and an associated per sample depth map are considered, we get • Multi-view video plus depth representation. Fig 2. – Test Sequence –Balloons with depth map [3]

  5. Fig 3. – Overview of the system structure and data format for the transmission of 3D video [12]

  6. MVC for MULTI-VIEW PLUS DEPTH • Encoding and decoding of each view of a multi-view data set separately ,referred to as simulcast coding , can be done with any video codec including H.264/AVC and HEVC. • This would be simple but inefficient as inter-view statistical dependencies are not exploited. • In order to exploit all the statistical dependencies within multi-view dataset, inter-view prediction has to be combined with temporal prediction. • As seen in fig 3b. , In MVC, one of the views is conventionally coded in conformance to the HEVC codec. For coding the other views, in addition to previously coded pictures of the same view already coded co-located pictures of other views can also be used as reference pictures. • In Multi –view video plus depth (MVD) format, only a few views are actually coded. Based on the transmitted videos and depth maps, additional views can be rendered.. Fig 4b. – Multi-view coding structure with hierarchical B pictures for both temporal (black arrows) and inter-view prediction(red arrows) [2] Fig 4a. – Simulcast coding structure with hierarchical B pictures for temporal prediction(black arrows)[2]

  7. Basic 3d video codec structure Fig 5. – Block Diagram of a 3D Video Codec[4]

  8. Mvd codec- working • The basic structure of the 3D video codec is shown in the block diagram of Figure 5. In • principle, each component signal is coded using an HEVC-based codec. The resulting • bit stream packets, or more accurately, the resulting Network Abstraction Layer (NAL) • units, are multiplexed to form the 3D video bit stream. • The base or independent view is coded using an unmodified HEVC codec. The base view • sub-stream can be directly decoded using the conventional HEVC decoder. • For coding the dependent views and the depth data, modified HEVC codec are used, • which are extended by including additional coding tools and inter-component prediction • techniques that employ already coded data inside the same access unit as indicated by • the red arrows in Figure 5. • For enabling an optional discarding of depth data from the bit stream, e.g., for • supporting the decoding of a stereo video suitable for conventional stereo displays, • the inter-component prediction can be configured in a way that video pictures can • be decoded independently of the depth data..

  9. MVD- CODING ALGORITHM • The video pictures and, when present, the depth maps are coded access unit by access unit, as it is illustrated • in Figure 6. • An access unit includes all video pictures and depth maps that correspond to the same time instant. NAL • units containing camera parameters may be additionally associated with an access unit. • The video pictures and depth maps corresponding to a particular camera position are indicated by a view • identifier (viewId). All video pictures and depth maps that belong to the same camera position are associated • with the same value of viewId. • Inside an access unit, the video picture and, when present, the associated depth map with viewId equal to 0 are • coded first, followed by the video picture and depth map with viewId equal to 1, etc. • For ordering the reconstructed video pictures and depth map after decoding, each value of viewId is associated • with another identifier called view order index (VOI). The view order index is a signed integer values, which specifies • the ordering of the coded views from left to right. Fig 6. - Access units structure and coding order of view components[12]

  10. COMPARSION – MVD AND HEVC CODEC • CODING OF DEPENDENT VIEWS -- Additional tools have been integrated into the HEVC • codec, which employ already coded data in other views for efficiently representing a • dependent view. These tools include - Disparity-compensated prediction, • View synthesis based inter-view prediction, Post processing in-loop filtering, • Inter-view motion prediction, Depth-based motion parameter prediction, • Inter-view residual prediction, Adjustment of QP of texture based on depth data. • CODING OF DEPTH MAPS – There are certain additional tools and also some tools are • removed for coding of Depth maps. Some of the differences are -- Depth Maps are • coded in 4:0:0 format, Non-linear depth representation is used, Z-near Z-far • compensated weighted prediction, Modified motion compensation and motion vector • coding ( No interpolation is used i.e. for depth maps, the inter-picture prediction is • always performed with full-sample accuracy. Disabling of in-loop filtering ( deblocking • filter and SAO), Depth modeling modes ( Four new Intra-prediction modes are used), • Motion parameter inheritance.

  11. THE PROPOSAL • The aim of the project is to import the MVD coding tools into the HM 9.1. • The project will be carried out in the following steps: • Learn about the tools that are different to MVD when compared to • standard HEVC codec. • Study and compare the MVD extensions of HEVC carried out on HM 5.1 • by Fraunhoffer HHI and Qualcomm ( the two most recent ones). Import the MVD to HM 9.1 based on the observations from the first two • stages. • Present the results and report the changes in bitrates and PSNR between the • new imported software and the previously studied

  12. INTERIM RESULTS • The sequence balloons (1024 x 768) has been used. The tested configuration is the • balloons sequence with three camera view including depth with QP of 30. • The encoding is done using both HTM 5.1 and HTM 6.0 to compare them on this • system. • The results are as shown in the next few slides,

  13. INTERIM RESULTS – HTM 5.1 v/s htm 6.0

  14. INTERIM RESULTS – HTM 5.1 v/s htm 6.0 BIT-rate

  15. INTERIM RESULTS – HTM 5.1 v/s htm 6.0 Y-PSNR

  16. Video used – before and after compression – htm 5.1 Artifacts

  17. Video used – before and after compression – htm 6.0 Artifacts Artifacts

  18. Other work done • Importing the 3D compression features into the HM 9.2 has been started. • Unfortunately, results can only be shown once complete porting of the encoder • is complete. • HTM 6.0 is referred when porting the 3D compression techniques to 9.2

  19. AVC: Advanced video coding ITU: International telecommunication union MPEG: Moving picture experts group MVC: Multi-View Coding MVD: Multi-View plus Depth MVV: Multi-View Video SAO: Sample Adaptive Offset VCEG: Video coding experts group VOI: View Order Index List of acronyms

  20. [1] Overview of the High Efficiency Video Coding (HEVC) Standard Gary J. Sullivan, Fellow, IEEE, Jens-Rainer Ohm, Member, IEEE, Woo-Jin Han, Member, IEEE, and Thomas Wiegand, Fellow, IEEE. [2] P. Merkle, A Smolic, K. Müller, and T. Wiegand, “Multi-View video plus depth data representation and coding”. PCS 2007 [3] http://www.tanimoto.nuee.nagoya-u.ac.jp/~fukushima/mpegftv/ [4] H. Schwarz, C. Bartnik, S. Bosse, H. Brust, T. Hinz, H. Lakshman, D. Marpe, P. Merkle, K. Müller, H. Rhee, G. Tech, M. Winken, and T. Wiegand, "3D Video Coding Using Advanced Prediction, Depth Modeling, and Encoder Control Methods", Picture Coding Symposium, May 2012. [5] G. Tech, H. Schwarz, K. Müller, and T. Wiegand, "Effects of synthesized View Distortion based 3D Video Coding on the Quality of interpolated and extrapolated Views", IEEE Intl. Conf. on Multimedia and Exposition, July 2012. [6] P. Merkle, C. Bartnik, K. Müller, D. Marpe, and T. Wiegand, "3D Video: Depth Coding Based on Inter-component Prediction of Block Partitions", Picture Coding Symposium, May 2012. [7] H. Schwarz and T. Wiegand, "Inter-View Prediction of Motion Data in Multiview Video Coding", Picture Coding Symposium, May 2012. [8] G. Tech, H. Schwarz, K. Müller, and T. Wiegand, "3D Video Coding using the Synthesized View Distortion Change", Picture Coding Symposium, May 2012. [9] M. Winken, H. Schwarz, and T. Wiegand, "Motion Vector Inheritance for High Efficiency 3D Video plus Depth Coding," Picture Coding Symposium, May 2012. [10] S. Bosse, H. Schwarz, T. Hinz, T. Wiegand, "Encoder Control for Renderable Regions in High Efficiency Multiview Video Plus Depth Coding", Picture Coding Symposium, May 2012. [11] hevc.info [12] “Test Model under Consideration for HEVC based 3D video coding”, ISO/IEC JTC1/SC29/WG11 MPEG2011/N12559 February 2012, San Jose, CA, USA References

More Related