Video Coding

1. Chapter III Video Coding By Theerayod Wiangtong, PhD., DIC Mahanakorn University of Technology

2. Part I: Introduction

3. What is “Video”? • General speaking, “Video” refers to pictorial (visual) info, including still images & time varying images (aka image sequences) • In most references, “video” refers to time-varying images unless otherwise is specified

4. Video Signal

5. Digital Video • What is digital video? It refers to the capturing, manipulation and storage of video in digital formats. A digital video (DV) camcorder, for example, is a video camera that captures and stores images on a digital medium. • What is different than analog video? • The way the signal is transmitted or stored • Every picture element (pixel) is represented by a sequence of 0’s and 1’s

6. How is it defined by? • Frame rate • Vertical resolution • Horizontal resolution • Bit-depth

7. What is so special about it? • Less subject to noise • Better quality than analog • Computer-friendly • Fast, small, reusable media, rather than videotapes. • Allows advanced editing and processing • Allows repeated reproduction without losing quality • Allows better compression and encryption schemes

8. How to obtain it? • Directly using a digital video camera • Obtaining it from an analog source (sampling and quantizing a raster scan)

9. Color Models • In the RGB color model, the three colors are equally important • The human visual system is less sensitive to color than to luminance (brightness) • The luminance/chrominance representation reduces the bandwidth requirements of the video signal • It ensures the compatibility with monochrome TV systems • YCbCr is the adopted color model in most of the video coding standards

10. Digital Video Compression • What? Reducing the quantity of data used to represent video content without excessively reducing the quality of the picture. It also reduces the number of bits required to store and/or transmit digital media. • Why? temporal 300fps 30fps 1M 10M spatial 8bpp 32bpp Bit-depth

11. The Need for Video Compression • High-Definition Television (HDTV) • 1920x1080 • 30 frames per second (full motion) • 8 bits for each three primary colors (RGB) Total 1.5 Gb/sec! • Cable TV: each cable channel is 6 MHz • Max data rate of 19.2 Mb/sec • Reduced to 18 Mb/sec w/audio + control … Compression rate must be ~ 80:1!

12. Video Coding Encoded Video “Bitstream” Encoder Storage or Transmission Medium Decoder Reconstructed Video Original Video

13. Layers of DTV Standard Transmitter Process Receiver Process PICTURE FORMATS Production Display COMPRESSION Compression Decompression TRANSPORT Packetizing Depacketizing TRANSMISSION Modulation Demodulation

14. Typical Video Sequences

15. QCIF CIF HD 720p SD 4CIF Typical Video Sequences

16. Typical Transmission andStorage Capacity

17. Subjective Video Quality • A subjective characteristic of video quality. It is concerned with how video is perceived by a viewer and designates his or her opinion on a particular video sequence.

18. Subjective Video Quality • Mathematical models that successfully emulate the subjective quality assessment results, based on criteria and metrics that can be measured objectively.

19. Better Performance R-D Performance

20. Part II: Video Coders

21. Video Compression Fundamental • Compression using different profiles and levels through • Spatial redundancy • Temporal redundancy • A profile defines a set of coding tools or algorithms that can be used in generating a compliant bitstream. • Within each profile, there are a number of levels designed for a wide range of applications, bitrates, resolutions, qualities, and services. A level places constraints on certain key parameters of the bitstream.

22. Levels and Profiles

23. Types of Video Redundancy • Spatial Intra-frame Coding • Temporal Inter-frame Coding • Psycho-visual Use less bits for chroma • Statistical Entropy Coding

24. Spatial Redundancy • Take advantage of similarity among most neighboring pixels

25. Loss of Resolution Original (63 kb) Low (7kb) Very Low (4 kb)

26. 950 951 952 Temporal Redundancy • Take advantage of similarity between successive frames

27. Chroma Sub-sampling

28. Types of Coding Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Coding Motion Compensated Predictive (MCP) Coding Object-based and scalable video coding Motion segmentation, scalability issues 28

29. A Tour of JPEG Coding Standard Components • Transform -8×8 DCT -boundary padding • Quantization -uniform quantization -DC/AC coefficients • Coding -Zigzag scan -run length/Huffman coding 29

30. Quantization • major reduction • controls ‘quality’ JPEG: Spatial Redundancy Reduction “Intra-Frame Encoded” Zig-Zag Scan, Run-length coding

31. Step 1: Transform • DC level shifting -128 • 2D DCT DCT 31

32. Step 2: Quantization Why increase from top-left to bottom-right? Q-table Q 32

33. Step 3: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) End Of the Block: All following coefficients are zero Zigzag Scan 33

34. Entropy Coding (e.g.) • Mainly concerned with representation of information

35. Entropy Coding • Lavg,1 = 3 bits per symbol • Lavg,2 = 4x0.1+2x0.2+1x0.5+4x0.05+3x0.15 = 1.95 bits per symbol • Code 2 with variable-length coding is more efficient than code 1 with natural binary coding.

36. Importance of Coeff’s

37. Types of Coding Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Coding Motion Compensated Predictive (MCP) Coding Object-based and scalable video coding Motion segmentation, scalability issues 37

38. Conditional Replenishment Based on motion detection rather than motion estimation Partition the current frame into “still areas” and “moving areas” Replenishment is applied to moving regions only Repetition is applied to still regions Need to transmit the location of moving areas as well as new (replenishment) information No motion vectors transmitted 38

39. Conditional Replenishment 39

40. Motion Detection 40

41. From Replenishment to Prediction A more powerful approach of exploiting temporal dependency is prediction Locate the best match from the previous frame Use the history to predict the current

42. Motion-Compensated Predictive Coding Residual and Motion vector

43. A Closer Look Frame n-1 Frame n

44. MCP Key Components • Motion Estimation/Compensation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size 44

45. Motion Estimation and Compensation • Predict the current frame based on reference frames while compensating for the motion. • For each motion compensation block • Find the block in the reference decoded frame that gives the least distortion. • If the distortion is too high then code the block independently. (intra block) • Otherwise code the difference (inter block)

46. Motion Compensation If motion estimation was effective – little data left in difference macroblock – more efficient compression.

47. MCP Key Components • Motion Estimation/Compensation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size

48. Motion Vector

49. Block-based Motion Model Block size Fixed vs. variable Motion accuracy Integer-pel vs. fractional-pel Number of hypothesis Overlapped Block Motion Compensation (OBMC) Multi-frame prediction 49

50. Block-based Motion Model