City University of Hong Kong

1. City University of Hong Kong INTRODUCTION TO MPEG Moving Picture Expert Group - Established in 1988 by the Joint ISO/IEC Technical Committee on IT. Mission - To develop standards for coded representation of motion pictures and audio at a bit rate of up to 1.5Mb/s. MPEG-1 was issued in 1992. MPEG-2 (1994) - higher quality (not lower than NTSC and PAL) with bit rates between 2-10Mb/s. Applications - Digital CATV and Terrestrial digital broadcasting distribution, Video recording and retrieval.

2. City University of Hong Kong BASIS OF MPEG ENCODING Lossy compression Trade off image quality with bit rate according to objective or subjective criteria Video sequences usually contains large statistical redundancies in both temporal and spatial directions Intraframe coding Interframe coding Subsampling of Chrominance - Human eye is more sensitive to luminance than chrominance

3. City University of Hong Kong Intraframe Coding Encoding of a single picture Similar to JPEG Discrete Cosine Transform- Converts spatial to frequency domain Quantization of spectral coefficients DPCM to encode DC terms Zigzag scan to group zeros into long sequences, followed by run-length coding Lossless, Variable Length Coding to encode AC coefficients

4. City University of Hong Kong Interframe Coding Remove temporal redundancies between frames Use extensively in MPEG-1 and MPEG-2 Based on estimation of motion between video frames Use of motion vectors to describe displacement of pixels from one frame to the next Spatial correlation between motion vectors are high One motion vector can represent the motion of a block of pixels.

5. Previous frame Current frame City University of Hong Kong Intraframe Coding For each image block in the current frame, Find its nearest counterpart in the previous frame. Record the displacement vector Figure 1

6. City University of Hong Kong Interframe Coding Previous Block Current Block Search Location Location Window mv Figure 2 Frame N-1 Frame N Only the prediction error (residual) images are transmitted Good prediction reduces information content in residual images

7. Block Search in MPEG Partition the previous and the current images into non-overlapping square blocks of size NxN Previous frame Current frame

8. Block Search in MPEG Partition the previous and the current images into non-overlapping square blocks of size NxN Previous frame Current frame e.g., N=8

9. Block Search in MPEG Partition the previous and the current images into non-overlapping square blocks of size NxN Previous frame Current frame Represent each block with a 2D matrix: f(x,y) for previous frame g(x,y) for current frame

10. Block Search in MPEG Partition the previous and the current images into non-overlapping square blocks of size NxN Previous frame Current frame Represent each block with a 2D matrix: f(x,y) for previous frame g(x,y) for current frame Difference between any two blocks is given by

11. Block Search in MPEG Partition the previous and the current images into non-overlapping square blocks of size NxN Previous frame Current frame Represent each block with a 2D matrix: f(x,y) for previous frame g(x,y) for current frame The lower the difference, the more similar is the pair of blocks

12. Block Search in MPEG A motion vector is computed for ‘EVERY’ blocks in the current frame. HOW? Previous frame Current frame

13. Block Search in MPEG A motion vector is computed for ‘EVERY’ blocks in the current frame. HOW? Previous frame Current frame Each block in the current frame matched against all the blocks in the previous frame, the closest one is taken to be its counterpart.

14. Block Search in MPEG MV=(-2,-3) x y Previous frame Current frame Each block in the current frame matched against all the blocks in the previous frame, the closest one is taken to be its counterpart.

15. Block Search in MPEG MV=(-1,-3) x y Previous frame Current frame Each block in the current frame matched against all the blocks in the previous frame, the closest one is taken to be its counterpart.

16. Block Search in MPEG MV=(-1,-2) x y Previous frame Current frame Each block in the current frame matched against all the blocks in the previous frame, the closest one is taken to be its counterpart.

17. Block Search in MPEG Previous frame Current frame The method is slow, especially if the image resolution and N are large.

18. Simplified Block Search in MPEG A motion vector is computed for ‘EVERY’ blocks in the near neighborhood of the current frame. Previous frame Current frame For example, only the blocks that are adjacent to the current one is tested. The method is faster but the search area is restricted

19. Simplified Block Search in MPEG A motion vector is computed for ‘EVERY’ blocks in the near neighborhood of the current frame. Search Window Previous frame Current frame For example, only the blocks that are adjacent to the current one is tested. The method is faster but the search area is restricted

20. Simplified Block Search in MPEG A motion vector is computed for ‘EVERY’ blocks in the near neighborhood of the current frame. Search Window Previous frame Current frame Assumption: changes between frames are small and are restricted within the search window. However, the search time is still long

21. Non-optimal Block Search in MPEG Given a block in the current frame, search the best match in the previous frame along the vertical direction Search Window Previous frame Current frame

22. Non-optimal Block Search in MPEG Given a block in the current frame, search the best match in the previous frame along the vertical direction Search Window Best Match Previous frame Current frame

23. Non-optimal Block Search in MPEG Search the best match in the previous frame along the horizontal direction Search Window Solution Previous frame Current frame

24. Non-optimal Block Search in MPEG Search the best match in the previous frame along the horizontal direction Search Window Solution Previous frame Current frame Non-optimal solution with the assumption of smooth intensity distribution

25. City University of Hong Kong MPEG I • Applications - multimedia and video transmission • Based on JPEG and H.261 standards • Flexible picture size and frame rate specified by users • Video source - Non-interlaced video signals. • Minimum requirements on decoders • resolution of 720X576 • 30 frames/s • 1.86Mb/s

26. City University of Hong Kong MPEG bit stream hierarchy Layer structure in MPEG bitstream • Sequence • Group Of Pictures (GOP) • Picture • Slice • Macroblock • Block

27. City University of Hong Kong MPEG bit stream hierarchy Video Sequence Group Of Pictures A Picture A Slice Macroblock Block

28. City University of Hong Kong MPEG I Encoding Scheme - Six major steps • Partitioning of images into Macroblocks (MB) • Intraframe coding on one out of every K images • Motion estimation on MBs • Generate (K-1) predicted frames • Encode residual error images • Conditional Replenishment of Macroblocks

29. City University of Hong Kong MPEG I Encoding Scheme - Image Format • An image is partitioned into Macroblocks of size 16X16 • 1 MB = 4 luminance (Y) and 2 chrominance blocks (U,V) • The sampling ratio between Y, U and V is 4:1:1

30. MPEG I Encoding Scheme - Image Format Figure 3

31. MPEG I Encoding Scheme - Chrominance subsampling Chrominance format: 4:4:4, 4:2:2, 4:1:1, 4:2:0 Assuming 8bits for Y, U and V components 4:4:4 4*8 (Y) + 4*8 (U) + 4*8 (V) = 96 bits Bits per pixel = 96/4 = 24 bpp

32. MPEG I Encoding Scheme - Chrominance subsampling Chrominance format: 4:4:4, 4:2:2, 4:1:1, 4:2:0 Assuming 8bits for Y, U and V components 4:2:2 4*8 (Y) + 2*8 (U) + 2*8 (V) = 64 bits Bits per pixel = 64/4 = 16 bpp

33. MPEG I Encoding Scheme - Chrominance subsampling Chrominance format: 4:4:4, 4:2:2, 4:1:1, 4:2:0 Assuming 8bits for Y, U and V components 4:1:1 4*8 (Y) + 1*8 (U) + 1*8 (V) = 48 bits Bits per pixel = 48/4 = 12 bpp

34. MPEG I Encoding Scheme - Chrominance subsampling Chrominance format: 4:4:4, 4:2:2, 4:1:1, 4:2:0 Assuming 8bits for Y, U and V components 4:2:0 4*8 (Y) + 1*8 (U) + 1*8 (V) = 48 bits Bits per pixel = 48/4 = 12 bpp

35. City University of Hong Kong MPEG I Encoding Scheme - Interframe Coding • DCT • Weighted (I-frame)/Uniform (P-frame) Quantization • DPCM on DC terms • Zigzag scan + runlength + VLC

36. MPEG I Encoding Scheme - Interframe Coding Block to be Macroblock encoded sz: Step Size DCT 8 sz 8 DC JPEG encoded DC Q DPCM 8 8 AC ZigZag Scanning Runlength JPEG encoded AC VLC Encoding Figure 4

37. MPEG I Encoding Scheme - Interframe Coding • Previous I or P frame is stored in both encoder and decoder • Motion Compensation is performed on a macroblock basis • One motion vector (mv) is generated for each macroblock • The mvs are coded and transmitted to the receiver

38. MPEG I Encoding Scheme - Interframe Coding • Motion prediction error of pixels in each macroblock is calculated • Error blocks (size 8X8) are encoded in the same manner as those in the I-Picture • A video buffer plus step size adjustment maintain a constant target bit-rate

39. A closed-loop Predictor • Current signal x(n) is predicted from previous sample x(n-1). Predicted value is xp(n) • Predicted error e(n)=x(n)-xp(n) is compressed (encode) and transmitted • The encoded error is decoded and added back to xp(n) to reconstruct the current signal x(n). However there are loss in the codec and the reconstructed signal xr(n)is not identical to x(n) • xr(n) is taken to predicted the next sample x(n+1)

40. City University of Hong Kong MPEG I Encoding Scheme - Interframe Coding Block to be CONTROL ENCODED encoded RESIDUAL sz ERROR RLC + DCT Q VB 8 VLC - 8 -1 Q -1 DCT + + MOTION MC + FRAME VECTOR STORE Figure 5

41. City University of Hong Kong MPEG I Encoding Scheme - Bi-directional Prediction • I-Pictures are encoded independently • I-Pictures can therefore be used as access point for random access, fast-forward (FF) or fast-reverse (FR) • P-Pictures cannot be decoded alone, hence cannot be used as an access point • B-Pictures are constructed with the nearest I or P Pictures • Backward prediction requires the presence of the start and end frames, both can be used as access points

42. 2 1 3 4 6 7 5 MPEG I Encoding Scheme - Random Access I B B P B B P Figure 6a Order of Coding

43. MPEG I Encoding Scheme - Random Access I I I I I I I I P P I P P I Figure 6b I P B I P B I Compression Random Access Coding Delay I Pictures only Low Highest Low I and P Pictures Medium Low Medium I, P and B Pictures High Medium High

44. City University of Hong Kong MPEG I Encoding Scheme - Conditional Replenishment • Only macroblocks that have been changed in the decoder are updated • Three types of MB are classified in MPEG standard • Skipped MB - Zero motion vector, the MB is neither encoded nor transmitted • Inter MB - Motion Prediction is valid, the MB type and address, motion vector and the coded DCT coefficients are transmitted • Intra MB - Encoded DCT coefficients of the MB are transmitted. No Motion Compensation is used

45. Macroblock type in P pictures Q Pred-mcq C No Q Pred-mc Non I No C Pred-m MC Q Intra-q I No Q MB Intra-d Q Pred-cq C No Q No MC Pred-c No C Skipped

46. Macroblock type in P pictures Non-zero motion vector, Error coded with defined quantization Q Pred-mcq C No Q Pred-mc Non I No C Pred-m MC Q Intra-q I No Q MB Intra-d Q Pred-cq C No Q No MC Pred-c No C Skipped

47. Macroblock type in P pictures Q Pred-mcq Non-zero motion vector, Error coded with default quantization C No Q Pred-mc Non I No C Pred-m MC Q Intra-q I No Q MB Intra-d Q Pred-cq C No Q No MC Pred-c No C Skipped

48. Macroblock type in P pictures Q Pred-mcq C Non-zero motion vector, Error not coded No Q Pred-mc Non I No C Pred-m MC Q Intra-q I No Q MB Intra-d Q Pred-cq C No Q No MC Pred-c No C Skipped

49. Macroblock type in P pictures Q Pred-mcq C No Q Pred-mc Non I No C Pred-m MC Macroblock intra-coded with defined quantization Q Intra-q I No Q MB Intra-d Q Pred-cq C No Q No MC Pred-c No C Skipped

50. Macroblock type in P pictures Q Pred-mcq C No Q Pred-mc Non I No C Pred-m MC Q Intra-q I Macroblock intra-coded with default quantization No Q MB Intra-d Q Pred-cq C No Q No MC Pred-c No C Skipped