ENEE 631 Project Video Codec and Shot Segmentation

1. ENEE 631 Project Video Codec and Shot Segmentation Aravind Sundaresan Vikas Raykar ENEE 631 Project: Video Codec

2. Main features/ functionality • Can encode monochrome video with frame dimensions (16*M, 16*N). • Codes the sequence as a series of I/P frames. The I/P decision is made according to suitability of each method. (Example: when a scene change is detected, the subsequent frame is coded INTRA). • The frames are periodically coded as INTRA according to the INTRA refresh rate parameter. • Temporal prediction is closed loop. Performs full-pel Motion Estimation in a window of of dimensions 48 x 48. Macroblocks within an I-frame are coded as INTRA/ INTER according to the compression achieved. • Has a resynchronization marker at frame level. ENEE 631 Project: Video Codec

3. Video Codec Structure • The Video Codec is split into two programs Encoder and Decoder. Both of them have three layers. • Top Layer: Takes care of interface and I/O. • Performs the necessary Initializations. • Splits the input into frames and feeds them to the next layer sequentially. ENEE 631 Project: Video Codec

4. Input Top Layer Raw frame Recon. frame ME, MC Blocks Intermediate Layer Residue frame Recon. Residue frame Encoder Blocks Bottom Layer Bitstream Video Encoder - Block diagram ENEE 631 Project: Video Codec

5. Video Codec Structure • Intermediate Layer. This Layer performs the frame level manipulations and also takes care of the frame-level and macroblock-level decision making in the encoder. • Performs Motion Estimation and Compensation or removes 128 from the frame to get Residue Frame. • Feeds the residue frame to the frame encode layer. • The reconstructed residue frame is used to reconstruct the current frame for future prediction. • Bottom Layer: This layer performs the actual coding. • A hybrid coding technique, that employs both predictive coding to remove temporal redundancy and transform coding to remove spatial redundancy is used. • The frame is split into macroblocks and each macroblock is coded separately. The bits generated are put in the bitstream. ENEE 631 Project: Video Codec

6. Top Layer (Interface Layer) • Performs the necessary initializations (such as Loading Huffman tables). • Serves as interface between user and the actual encoder. • Input sequence is read and passed as frames to the lower layer. • Very first frame is forced to be INTRA. Subsequent frames are by default directed to be coded as INTER. The lower layer may dynamically decide to code such a frame AS INTRA according to various parameters (scene change / INTRA refresh). ENEE 631 Project: Video Codec

7. Intermediate Layer (Control Layer) • This is an important layer in that most of the decision making is performed here. These decisions are aimed at selecting the best coding technique according to the input frame. The decisions made and the functions performed are listed below. • Intra / Inter Decision. The top layer has the ability to force the Intra Option. The Intermediate layer has the option to change the INTER option to INTRA. If the number of consecutive frames coded as P-frames equals a certain parameter (INTRA REFRESH RATE), the next frame is coded as an I-frame. ENEE 631 Project: Video Codec

8. Intermediate Layer (Control Layer) • Motion Estimation and Compensation. In case of Intra Macroblocks, 128 is subtracted from the Macroblock. Based on the output of the Motion Estimation a decision is made whether to code the frame as INTER. • The frame to be coded is split into a MC frame and Residue frame (in case of INTRA frame, the MC frame comprises of pixels with value 128). • The Residue frame is passed to the Encode frame layer. The layer returns the reconstructed residue frame which is added to the MC frame to be used for future prediction. (Closed Loop prediction to avoid error accumulation) ENEE 631 Project: Video Codec

9. Bottom Layer (Encoder Layer) • The frame is split into Macroblocks each of which comprises of 4 blocks. The macroblocks are read in raster scan order and coded sequentially and the blocks in the Macroblock are also similarly coded. Each macroblock consists of a header followed by the coefficient data. The header contains • Coded Information (Coded/ Not coded, INTRA/ INTER, etc) • Motion Vector (for INTRA MBs) • Coded Block Pattern • Optionally include the Quantizer (or differential Quantizer). ENEE 631 Project: Video Codec

10. Bottom Layer (Encoder Layer) • The coding Procedure is described below. • Each block is transform coded using the DCT. The DCT coefficients are quantized. The Quantization tables for INTRA and INTER blocks are different. • Resulting matrix is split into DC and AC coefficients which are scanned in a zigzag manner and coded using fixed Huffman tables and run-length coding techniques. • For run-length values not found in the table the run-length and level values are coded using an ESCAPE code and fixed length codes. If none of the blocks contain any coefficients, the MB is 'not coded'. • If only some of the blocks are coded, the corresponding bit is set in the CBP. ENEE 631 Project: Video Codec

11. Output Top Layer Ref. frame Cur. Recon. frame MC Blocks Intermediate Layer Recon. Residue frame Decoder Blocks Bottom Layer Bitstream Video Decoder - Block diagram ENEE 631 Project: Video Codec

12. Decoded Frames ENEE 631 Project: Video Codec

13. Results • 30 frames encoded • Size of compressed stream = 434096 bytes • Size of 30 frames: 30.240.352 = 2534400 bytes • Compression = 17% • Greater Compression can be achieved by • Increasing quantizer step • Increase ratio of P frames to I frames (current ratio 10:1) ENEE 631 Project: Video Codec