Overview of the H.264/AVC Video Coding Standard

1. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 13, NO. 7, JULY 2003 Overview of the H.264/AVC Video Coding Standard ThomasWiegand, Gary J. Sullivan, Gisle Bjøntegaard, and Ajay Luthra

2. Outline • Overview of the technical features of H.264/AVC • Profiles and Levels

3. Goals of the H.264/AVC • Video Coding Experts Group (VCEG), ITU-T SG16 Q.6 • H.26L project (early 1998) • Target –double the coding efficiency in comparison to any other existing video coding standards for a broad variety applications. • H.261, H.262 (MPEG-2), • H.263 (H.263+, H.263++)

4. Scope of video coding standardization Pre-Processing Encoding Source Post-Processing & Error recovery Decoding Destination Scope of Standard

5. Applications on H.264/AVC standard • Broadcast over cable, satellite, cable modem, DSL, terrestrial, etc. • Interactive or serial storage on optical and magnetic devices, DVD, etc. • Conversational services over ISDN, Ethernet, LAN, DSL, wireless and mobile networks, modems, etc. or mixtures of these. • Video-on-demand or multimedia streaming services over ISDN, cable modem, DSL, LAN, wireless networks, etc. • Multimedia messaging services (MMS) over ISDN, DSL, ethernet, LAN, wireless and mobile networks, etc.

6. Structure of H.264/AVC video encoder Video Coding Layer Control Data Coded Macroblock Data Partitioning Coded Slice/Partition Network Abstraction Layer H.320 MP4FF H.323/IP MPEG-2 Etc.

7. Design feature highlights (1) — improved on prediction methods • Variable block-size motion compensation with small block sizes • A minimum luma motion compensation block size as small as 4×4. • Quarter-sample-accurate motion compensation • First found in an advanced profile of the MPEG-4 Visual (part 2) standard, but further reduces the complexity of the interpolation processing compared to the prior design.

8. Design feature highlights (2)— improved on prediction methods • Motion vectors over picture boundaries • First found as an optional feature in H.263 is included in H.264/AVC. • Multiple reference picture motion compensation • Decoupling of referencing order from display order • (X)IBBPBBPBBP… => IPBBPBBPBB… • Bounded by a total memory capacity imposed to ensure decoding ability. • Enables removing the extradelay previously associated with bi-predictive coding.

9. Design feature highlights (3)— improved on prediction methods • Decoupling of picture representation methods frompicture referencing capability • Ｂ－framecould not be used asreferences for prediction • Referencing to closest pictures • Weighted prediction • A new innovation in H.264/AVC allows the motion-compensated prediction signal to be weighted and offset by amounts specified by the encoder. • For scene fading

10. Design feature highlights (4)— improved on prediction methods • Improved “skipped” and “direct” motion inference • Inferring motion in “skipped” areas => for global motion • Enhanced motion inference method for “direct”

11. Design feature highlights (5)— improved on prediction methods • Directional spatial prediction for intra coding • Allowing prediction from neighboring areas that were not coded using intra coding • Something not enabled when using the transform-domain prediction method found in H.263+ and MPEG-4 Visual

12. Design feature highlights (6)— improved on prediction methods • In-the-loop deblocking filtering • Building further on a concept from an optional feature of H.263+ • The deblocking filter in the H.264/AVC design is brought within the motion-compensated prediction loop

13. Design feature highlights (7)— other parts • Small block-size transform • The new H.264/AVC design is based primarily on a 4×4 transform. • Allowing the encoder to represent signals in a more locally-adaptive fashion, which reduces artifacts known colloquially as “ringing”.

14. Design feature highlights (8)— other parts • Hierarchical block transform • Using a hierarchical transform to extend the effective block size use for low-frequency chroma information to an 8×8 array • Allowing the encoder to select a special coding type for intra coding, enabling extension of the length of the luma transform for low-frequency information to a 16×16 block size

15. Design feature highlights (9)— other parts • Short word-length transform • While previous designs have generally required 32-bit processing, the H.264/AVC design requires only 16-bit arithmetic. • Exact-match inverse transform • Building on a path laid out as an optional feature in the H.263++ effort, H.264/AVC is the first standard to achieve exact equality of decoded video content from all decoders. • Integer transform

16. Design feature highlights (10)— other parts • Arithmetic entropy coding • While arithmetic coding was previously found as an optional feature of H.263, a more effective use of this technique is found in H.264/AVC to create a very powerful entropy coding method known as CABAC (context-adaptive binary arithmetic coding)

17. Design feature highlights (11)— other parts • Context-adaptive entropy coding • CAVLC (context-adaptive variable-length coding) • CABAC (context-adaptive binary arithmetic coding)

18. Design feature highlights (12)— Robustness to data errors/losses and flexibility for operation over a variety of network environments • Parameter set structure • The parameter set design provides for robust and efficient conveyance header information • NAL unit syntax structure • Each syntax structure in H.264/AVC is placed into a logical data packet called a NAL unit

19. Design feature highlights (13)— Robustness to data errors/losses and flexibility for operation over a variety of network environments • Flexible slice size • Unlike the rigid slice structure found in MPEG-2 (which reduces coding efficiency by increasing the quantity of header data and decreasing the effectiveness of prediction), slice sizes in H.264/AVC are highly flexible, as was the case earlier in MPEG-1.

20. Design feature highlights (14)— Robustness to data errors/losses and flexibility for operation over a variety of network environments • Flexible macroblock ordering (FMO) • Significantly enhance robustness to data losses by managing the spatial relationship between the regions that are coded in each slice • Arbitrary slice ordering (ASO) • sending and receiving the slices of the picture in any order relative to each other • first found in an optional part of H.263+ • can improve end-to-end delay in real-time applications, particularly when used on networks having out-of-order delivery behavior

21. Design feature highlights (15)— Robustness to data errors/losses and flexibility for operation over a variety of network environments • Redundant pictures • Enhance robustness to data loss • A new ability to allow an encoder to send redundant representations of regions of pictures

22. Design feature highlights (15)— Robustness to data errors/losses and flexibility for operation over a variety of network environments • Data Partitioning • Allows the syntax of each slice to be separated into up to threedifferent partitions for transmission, depending on a categorization of syntax elements • This part of the design builds further on a path taken in MPEG-4 Visual and in an optional part of H.263++. • The design is simplified by having a single syntax with partitioning of that same syntax controlled by a specified categorization of syntax elements.

23. Design feature highlights (16)— Robustness to data errors/losses and flexibility for operation over a variety of network environments • SP/SI synchronization/switching pictures • A new feature consisting of picture types that allow exact synchronization of the decoding process of some decoders with an ongoing video stream produced by other decoders without penalizing all decoders with the loss of efficiency resulting from sending an I picture • Enable switching a decoder between different data rates, recovery from data losses or errors, as well as enabling trick modes such as fast-forward, fast-reverse, etc.

24. NAL (Network Abstraction Layer) Video Coding Layer Control Data Coded Macroblock Data Partitioning Coded Slice/Partition Network Abstraction Layer H.320 MP4FF H.323/IP MPEG-2 Etc.

25. NAL (Network Abstraction Layer) • Designed in order to provide “network friendliness” • facilitates the ability to map H.264/AVC VCL data to transport layers such as: • RTP/IP for any kind of real-time wire-line and wireless Internet services (conversational and streaming); • File formats, e.g., ISO MP4 for storage and MMS; • H.32X for wireline and wireless conversational services; • MPEG-2 systems for broadcasting services, etc.

26. Key concepts of NAL • NAL Units • Byte stream and Packet format uses of NAL units • Parameter sets • Access units

27. NAL units 1 byte header payload Integer number of bytes Interleaved as necessary with emulation prevention bytes, which are bytes inserted with a specific value to prevent a particular pattern of data called a start code prefixfrom being accidentally generated inside the payload. The NAL unit structure definition specifies a generic format for use in both packet-oriented and bitstream-oriented transport systems, and a series of NAL units generated by an encoder is referred to as a NAL unit stream.

28. NAL units in byte-stream format use • E.g., H.320 and MPEG-2/H.222.0 systems • require delivery of the entire or partial NAL unit stream as an ordered stream of bytes or bits. • Each NAL unit is prefixed by a specific pattern of three bytes called a start code prefix. payload

29. NAL units in packet-transport system use • E.g., internet protocol/RTP systems • The inclusion of start code prefixes in the data would be a waste of data carrying capacity, so instead the NAL units can be carried in data packets withoutstart code prefixes. payload

30. VCL and no-VCL NAL units • VCL NAL units • The data that represents the values of the samples in the video pictures • Non-VCL NAL • Any associated additional information such as parameter sets(important header data that can apply to a large number of VCL NAL units) and supplemental enhancement information (timing information and other supplemental data that may enhance usability of the decoded video signal but are not necessary for decoding the values of the samples in the video pictures).

31. Parameter Sets (1) • A parameter set is supposed to contain information that is expected to rarely change and offers the decoding of a large number of VCL NAL units.

32. Parameter Sets (2) • Two types of parameter sets: • Sequence parameter sets • Apply to a series of consecutive coded video pictures called a coded video sequence; • Picture parameter sets • Apply to the decoding of one or more individual pictures within a coded video sequence.

33. Parameter Sets (3) The Structure VCL NAL unit Identifier to Picture parameter set Picture parameter set Sequence parameter set Identifier to Sequence parameter set Non VCL NAL unit

34. Parameter Sets (4) Transmission VCL NAL unit Non VCL NAL unit In-band VCL NAL unit Out of band Non VCL NAL unit

35. Parameter set use with reliable “out-of-band” parameter set exchange NAL unit with VCL Data encoded with PS#3 (address in Slice Header ) H.264/AVC Encoder H.264/AVC Decoder Reliable Parameter Set Exchange 1 2 3 3 2 1 • Parameter Set #3 • Video format PAL • Entr. Code CABAC • …

36. Access Units • A set of NAL units in a specified form is referred to as an access unit. start redundant coded picture access unit delimiter Supplemental Enhancement Information end of sequence SEI end of stream VCL NAL units slices or slice data partitions primary coded picture end

37. Coded Video Sequences • A coded video sequence consists of a series of access units that are sequential in the NAL unit stream and use only one sequence parameter set. • Can be decoded independently • Start with an instantaneous decoding refresh (IDR) –Intra picture. • A NAL unit stream may contain one or more coded video sequences.

38. VCL (Video Coding Layer) input video DCT Q VLC - output bitstream 16×16 macroblocks IQ Intra- Prediction IDCT Intra / inter Motion Compensation De-blocking Filter Motion Estimation Frame Memory output video Clipping Decoder YCbCr Color Space and 4:2:0 Sampling

39. Pictures, Frames, and Fields Bottom Field Progressive Frame Top Field ∆t Interlaced Frame (Top Field First)

40. Slices and Slice Groups (1) Slice #0 Slice #1 Slice #2 Subdivision of a picture into slices when not using FMO. (Flexible Macroblock Ordering)

41. Slices and Slice Groups (2) Slice Group #0 Slice Group #0 Slice Group #1 Slice Group #1 Slice Group #2 Subdivision of a QCIF frame into slices utilizing FMO.

42. Slice coding types • I Slice • P Slice • B Slice • SP Slice • Switching P slice • efficient switching between different pre-coded pictures becomes possible. • SI Slice • Switching I slice • Allowing an exact match of a macroblock in an SP slice for random access and error recovery purposes.

43. Adaptive Frame/Field Coding Operation • Three modes can be chosen adaptively for each frame in a sequence. • Frame mode • Field mode • Frame mode / Field coded • For a frames consists of mixed moving regions • The frame/field encoding decision can be made for each vertical pair of macroblocks (a 16×32 luma region) in a frame. • Macroblock-adaptive frame/field (MBAFF) Picture-adaptive frame/field (PAFF) 16% ~ 20% save over frame-only for ITU-R 601 “Canoa”, “Rugby”, etc.

44. Macroblock-adaptive frame/field (MBAFF) Top/Bottom Macroblocks in Field Mode A Pair of Macroblocks in Frame Mode

45. PAFF vs MBAFF • The main idea of MBAFF is to preserve as much spatial consistency as possible. • In MBAFF, one field cannot use the macroblocks in the other field of the same frame as a reference for motion prediction. • PAFF coding can be more efficient than MBAFF coding in the case of rapid global motion, scene change, or intra picture refresh. • MBAFF was reported to reduce bit rates14 ~ 16% over PAFF for ITU-R 601 (Mobile and Calendar, MPEG-4 World News)

46. Intra-Frame Prediction (1) • Intra 4×4 • Well suited for coding of parts of a picture with significant detail. • Intra_16×16 together with chroma prediction • More suited for coding very smooth areas of a picture. • 4 prediction modes • I_PCM • Bypassprediction and transform coding and send the values of the encoded samplesdirectly

47. Intra-Frame Prediction (2) • In H.263+ and MPEG-4 Visual • Intra prediction is conduced in the transform domain • In H.264/AVC • Intra prediction is always conducted in the spatial domain

48. Intra-Frame Prediction (3)

49. Intra-Frame Prediction (4) Across slice boundaries is not allowed.

50. Inter-Frame Prediction in P slices (1) Segmentations of the macroblock MB Types 8 8 16 8 8 16 8 8 16 16 8 8 8x8 Types 8 4 8 4 4 4 4 8 8 4 *P_Skip www.vcodex.com H.264 / MPEG-4 Part 10 : Inter Prediction