1 / 50

Roadmap

Roadmap. Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Motion Compensated Prediction (MCP) Object-based and scalable video coding* Motion segmentation, scalability issues. Introduction to Video Coding.

gannon
Download Presentation

Roadmap

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) • Motion Compensated Prediction (MCP) • Object-based and scalable video coding* • Motion segmentation, scalability issues EE591f Digital Video Processing

  2. Introduction to Video Coding • Lossless vs. lossy data compression • Source entropy H(X) • Rate-Distortion function R(D) or D(R) • Probabilistic modeling is at the heart of data compression • What is P(X) for video source X? • Modeling moving pictures is more difficult than modeling still images due to temporal dependency EE591f Digital Video Processing

  3. Shannon’s Picture Distortion Coder A Coder B Rate (bps) Which coder wins, A or B? EE591f Digital Video Processing

  4. Distortion Measures • Objective • Mean Square Error (MSE) • Peak Signal-to-Noise-Ratio (PSNR) • Measure the fidelity to original video • Subjective • Human Vision System (HVS) based • Emphasize visual quality rather than fidelity • We only discuss objective measures in this course EE591f Digital Video Processing

  5. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) • Motion Compensated Prediction (MCP) • Scalable video coding • 3D subband/wavelet coding and recent trend EE591f Digital Video Processing

  6. A Tour of JPEG Coding Standard Key Components • Transform -8×8 DCT -boundary padding • Quantization -uniform quantization -DC/AC coefficients • Coding -Zigzag scan -run length/Huffman coding EE591f Digital Video Processing

  7. JPEG Baseline Coder Tour Example EE591f Digital Video Processing

  8. Step 1: Transform • DC level shifting -128 • 2D DCT DCT EE591f Digital Video Processing

  9. Step 2: Quantization Why increase from top-left to bottom-right? Q-table Q EE591f Digital Video Processing

  10. Step 3: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) End Of the Block: All following coefficients are zero Zigzag Scan EE591f Digital Video Processing

  11. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) • Motion Compensated Prediction (MCP) • Scalable video coding • 3D subband/wavelet coding and recent trend EE591f Digital Video Processing

  12. Conditional Replenishment • Based on motion detection rather than motion estimation • Partition the current frame into “still areas” and “moving areas” • Replenishment is applied to moving regions only • Repetition is applied to still regions • Need to transmit the location of moving areas as well as new (replenishment) information • No motion vectors transmitted EE591f Digital Video Processing

  13. Conditional Replenishment EE591f Digital Video Processing

  14. Motion Detection EE591f Digital Video Processing

  15. From Replenishment to Prediction • Replenishment can be viewed as a degenerated case of prediction • Only zero motion vector is considered • Discard the history • A more powerful approach of exploiting temporal dependency is prediction • Locate the best match from the previous frame • Use the history to predict the current EE591f Digital Video Processing

  16. Differential Pulse Coded Modulation ^ xn-1 ^ ^ ^ yn yn yn xn xn _ Q + ^ ^ xn-1 xn D + D ^ xn-1 Decoder Encoder Xn,yn: unquantized samples and prediction residues ^ ^ Xn,yn: decoded samples and quantized prediction residues EE591f Digital Video Processing

  17. Motion-Compensated Predictive Coding EE591f Digital Video Processing

  18. A Closer Look EE591f Digital Video Processing

  19. Key Components • Motion Estimation/Compensation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE591f Digital Video Processing

  20. Block-based Motion Model • Block size • Fixed vs. variable • Motion accuracy • Integer-pel vs. fractional-pel • Number of hypothesis • Overlapped Block Motion Compensation (OBMC) • Multi-frame prediction EE591f Digital Video Processing

  21. Quadtree Representation of Motion Field with Variable Blocksize EE591f Digital Video Processing

  22. Rate-Distortion Optimized BMA Distortion alone Rate and Distortion counted bits using a VLC table EE591f Digital Video Processing

  23. Experimental Results Cited from G. Sullivan and L. Baker, “Rate-Distortion optimized motion compensation for video compression using fixed or variable size blocks”, Globecom’1991 EE591f Digital Video Processing

  24. Fractional-pel BMA • Recall the tradeoff between spending bits on motion and spending bits on MCP residues • Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically reduces the variance of MCP residues for some video sequence. • The gain quickly saturates as motion accuracy refines EE591f Digital Video Processing

  25. Example 8-by-8 block, integer-pel, var(e)=220.8 8-by-8 block, half-pel, var(e)=123.8 MCP residue comparison for the first two frames of Mobile sequence EE591f Digital Video Processing

  26. Fractional-pel MCP EE591f Digital Video Processing

  27. Multi-Hypothesis MCP • Using one block from one reference frame represents a single-hypothesis MCP • It is possible to formulate multiple hypothesis by considering • Overlapped blocks • More than one reference frame • Why multi-hypothesis? • The benefit of reducing variance of MCP residues outweighs the increased overhead on motion EE591f Digital Video Processing

  28. Example: B-frame fn-1 fn fn+1 EE591f Digital Video Processing

  29. Generalized B-frame fn-2 fn-1 fn+2 fn fn+1 EE591f Digital Video Processing

  30. Multi-Hypothesis MCP EE591f Digital Video Processing

  31. Key Components • Motion Estimation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE591f Digital Video Processing

  32. Motion Vector Coding • 2D lossless DPCM • Spatially (temporally) adjacent motion vectors are correlated • Use causal neighbors to predict the current one • Code Motion Vector Difference (MVD) instead of MVs • Entropy coding techniques • Variable length codes (VLC) • Arithmetic coding EE591f Digital Video Processing

  33. MVD Example MV1 MV2 MV MV3 Due to smoothness of MV field, MVD usually has a smaller variance than MV EE591f Digital Video Processing

  34. VLC Example MVx/MVy symbol codeword 0 1 1 010 1 2 -1 011 3 00100 4 2 -2 00101 5 3 00110 6 Exponential Golomb Codes: 0…01x…x m m EE591f Digital Video Processing

  35. Key Components • Motion Estimation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE591f Digital Video Processing

  36. MCP Residue Coding Transform Quantization Coding Conceptually similar to JPEG Transform: unitary transform Quantization: Deadzone quantization Coding: Run-length coding EE591f Digital Video Processing

  37. Transform Unitary matrix: A is real, A-1=AT Unitary transform: A is unitary, Y=AXAT Examples 8-by-8 DCT 4-by-4 integer transform EE591f Digital Video Processing

  38. Deadzone Quantization deadzone 2       0 codewords EE591f Digital Video Processing

  39. Key Components • Motion Estimation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE591f Digital Video Processing

  40. Lagrangian Multiplier Method Motion estimation Mode selection QUANT: a user-specified parameter controlling quantization stepsize EE591f Digital Video Processing

  41. Summary • How does MCP coding work? • The predictive model captures the slow-varying trend of the samples {fn} • The modeling of prediction residues {en} is easier than that of original samples {fn} • Fundamental weakness • Quantization error will propagate unless the memory of predictor is refreshed • Not suitable for scalable coding applications EE591f Digital Video Processing

  42. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) • Motion Compensated Prediction (MCP) • Scalable video coding • 3D subband/wavelet coding and recent trend EE591f Digital Video Processing

  43. Scalable vs. Multicast • What is scalable coding? foreman.yuv foreman.yuv foreman128k.cod foreman.cod foreman256k.cod foreman512k.cod foreman1024k.cod 128 256 512 1024 Multicast Scalable coding EE591f Digital Video Processing

  44. Spatial scalability EE591f Digital Video Processing

  45. Temporal scalability Frame 0,1,2,3,4,5,… Frame 0,4,8,12,… Frame 0,2,4,6,8,… 7.5Hz 15Hz 30Hz EE591f Digital Video Processing

  46. SNR (Rate) scalability PSNRavg=40dB PSNRavg=30dB PSNRavg=35dB PSNRi: PSNR of frame i EE591f Digital Video Processing

  47. Scalability via Bit-Plane Coding sign bit A=(a0+a12+a222+ … … +a727) Least Significant Bit (LSB) Most Significant Bit (MSB) Example A=129  sign=+,a0a1a2 …a7=10000001 sign=-, a0a1a2 …a7=00110011 A=-(4+8+64+128)=-204 EE591f Digital Video Processing

  48. Why DPCM Bad for Scalability? Frame number 3 1 2 … Base layer Ibase P P P Enhancement Layer 1 Ienh1 P P P Enhancement Layer 2 Ienh2 P P P suffer from drifting problem suffer from coding efficiency loss EE591f Digital Video Processing

  49. 3D Wavelet/Subband Coding y t x 2D spatial WT+1D temporal WT EE591f Digital Video Processing

  50. Motion-Adaptive 3D Wavelet Transform Recall Haar transform lifting-based implementation Motion-adaptive Haar transform W,W-1: forward and backward motion vector EE591f Digital Video Processing

More Related