Presents IMTC/Wainhouse Research European Forum 2003

1. Presents IMTC/Wainhouse Research European Forum 2003

2. H.264 Technology Update Presented by Stephan Wenger TU BerlinMember of the Joint Video Team

3. Outline • History of the H.264 Development • Current Timeline • Key Applications and Profiles • Technology in H.264 • Technology not included in H.264 • Royalty-free Baseline • News on Licensing • Performance IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

4. History of video compression • Early Video Codecs • Proprietary PCM, DPCM, ITU-T H.120 • Motion JPEG • First generation of DCT-based codecs • H.261, MPEG-1 Part 2, H.262/MPEG-2 Part 2 • 8x8 Block, 16x16 pel macroblock, MB-based motion compensation • Sophisticated interlace support (MPEG-2 only) • Second generation, H.263, H.263+, MPEG-4 Part 2 • Block-based motion compensation • Generally improved design • Error resilience features • Third generation, H.264, MPEG-4 Part 10 (AVC) • 4x4 Blocks for transform, flexible block sizes for motion compensation • Many new tools IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

5. History of H.264 • “Long-term” project H.26L was around since at least 1997 • Not much progress until finalization of H.263++ in 2000…... at which time the project got significant momentum • 09/01: core codec functionality (progressive scan, interactive applications) available • Very limited support for classic broadcast…… but interest was expressed in enhanced functionality for this market • No significant changes in the core codec since then • 12/01: MPEG adopts H.26L, formation of JVT • Support for broadcast requires tools for interlace, high-delay, buffer management • Mid 2002: “culture clashes”, unified codec design equally suitable for broadcast and interactive applications hammered out • Late 2002: finalization, debugging, profile design, politics • March 2003: standard out for ballot in the ITU-T and ISO/IEC IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

6. Current Timeline • In Short: It’s done! • 28.03.2003: H.264 is out for “Consent” in the ITU-T • Four sets of Comments have been received • Nothing critical • SG16 meeting in two weeks will likely ratify the Recommendation • In the ISO/IEC ballot has not yet started • Work is in progress on • Initial Corrigenda to catch those few bugs that slipped through the final editing process – may be rolled into the text if procedurally feasible • A small enhancement of H.264 to support 10 and 12 bit color and colorspaces other than YUV 4:2:0 • Conformance test points, Software availability, and similar • The author expects that JVT will dissolve in mid 2004 after the mentioned work items are completed IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

7. Key Applications and Profiles Baseline • Progressive Scan, Error Resilience • Conversational Services • Video-Conference, wireline/wireless Video-Telephone • Simple Streaming applications Main • Interlace, B-Slices, CABAC, NO Error Resilience • Broadcast Applications w/ MPEG-2 transport • DVD and similar, TV-quality Extended • Interlace, B-Slices, SP-Slices, Error Resilience • Streaming in wireline/wireless environments • Progressive Scan Broadcast? IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

8. H.264 Technology Common to all Profiles • 4x4 Transform, integer exact • Complex motion model, variable block sizes • Multiple Reference Frames, ERPS • Intra-Prediction (from Intra/Inter information) • Efficient entropy coding (CA-VLC) Baseline Main Extended ASO - ASO FMO - FMO RS - RS - Interlace, MB-AFF Interlace MB-AFF - CABAC - - B-Slices B-Slices - - SP-Slices Royalty Free Royalties Royalties IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

9. Non-H.264 Technology Considered, but not included • Scalability support – an academic concept • Temporal Scalability available through non-stored pictures • Spatial/SNR scalability never accepted by users in previous standards • FGS, no acceptance so far, drift-based • Reference Picture Re-Sampling (like Annex P/H.263) • Change picture sizes on-the-fly • Difficult to implement, no clear coding gain • Variable MB-sizes (like Annex Q/H.263) • No clear coding gain, flexible block sizes catch most of it) Considered and deferred • “Professional” Colorspaces (YUV 4:2:2, YUV 4:4:4, RGB) • Sampling Depth of 10 bits/12 bits IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

10. Royalty-Free Baseline • Royalty-Free Baseline (RFB) Goal established early • Common understanding in Videoconferencing Industry • Buy-in from most big players in the field • Very few exceptions • Goal was achieved by • Requiring commitment to RFB for all new proposals • Asking for commitments to RFB from existing patent holders on Baseline technology • Remove technology from Baseline where the patent holders were unwilling to sign on on the RFB • Extensive Studies were performed • Several $100,000 spent in lawyer fees • Extensive technical analysis – nothing evil is left Goal is achieved (so God will) IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

11. Licensing • Current and future patent holders to select Patent Agency • Several candidates (MPEG-LA, Dolby, at least two more) • Selection made by current (and future?) stakeholders • RFB support by patent agency is a key selection criterion • Baseline will be free, but one needs a license anyway • Evaluation cost has to be low for patents that are only in baseline • Licenses for consumer level product likely below $1.00 • To be competitive w/ MS Media 9 • Provisions for emerging markets, freeware, shareware • Minimum and maximum Caps to facilitate accounting • Likely the killer for MPEG licensing as we know it • MPEG-4: Killing a Zombie??? • Significantly cheaper than MPEG-2 part 2 IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

12. Performance Video Clip w/ the following characteristics • 640x480 resolution • TeleSuite’s professional, digital, progressive-scan, video-conferencing Camera • 30 fps • Shabby lens :-( • No pre/post-processing (!) • Single DSP implementation for the encoder (TI C6415, 600 MHz) • Encoder and Decoder from UB Video, optimized for high performance in a controlled environment (e.g. ERPS for background restore) • Real-time encoding/decoding (on 1.5 DSPs) • 300 kbit/s video bit rate, 384 kbit/s channel bit rate • Reconstructed Video encoded again using MPEG-4, 6 Mbit/s • So to use the Media Player for easier demonstration IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

13. IMTC/Wainhouse Research European Forum – May 2003 – Geneva, Switzerland

14. Thank You Stephan WengerTU Berlin stewe@cs.tu-berlin.de