Video As A Datatype

1. Video As A Datatype Ketan Mayer-Patel Comp 294-9 :: Fall 2003

2. The mothers of all video... • NTSC - American standard • PAL - European standard • SECAM - French standard (ugly stepchild) • For better or worse, this is where it all starts. Comp 294-9 :: Fall 2003

3. Color Spaces • One luminance component • Two chrominance components • Original TV was black and white. • Adding color had to be done in a compatible way. • NTSC: YIQ • PAL: YUV • In general: YUV and YCrCb used as terms. Comp 294-9 :: Fall 2003

4. YUV vs. RGB • Simple linear transform from one cube to the other. • Specific transform coefficients vary for specific systems, but a common one: • Y = 0.299 R + 0.587 G + 0.114 B • Cr = -0.147 R - 0.289 G + 0.437 B + 0.5 • Cb = 0.615 R - 0.515 G - 0.100 B + 0.5 Comp 294-9 :: Fall 2003

5. YUV Challenges • YUV is like taking RGB cube, standing it on a corner with (0,0,0) on bottom and (1,1,1) at top, rotating slightly, and taking the bounding box. • What problems might this incur? • Some RGB colors are illegal. • A lot of YUV colors are illegal. Comp 294-9 :: Fall 2003

6. A Note On Color • Previous view of color is EXTREMELY simplified. • Color is complicated • Frequency dependent response for contrast, lightness, etc. • Gamma correction.` Comp 294-9 :: Fall 2003

7. Scanning • Analog video signal is continuous voltage signal that gets scanned along the screen. • The electron stream controlled by two orthogonal sets of magnets. • Horizontal: Beam is moved from left to right and then quickly back. • Vertically: Beam is moved from top to bottom and then quickly back. Comp 294-9 :: Fall 2003

8. Scanning illustrated Voltage Time Comp 294-9 :: Fall 2003

9. Structure of Video • Blinn’s article. • Macrostructure • Frequency peaks at frame rate. • Microstructure • Frequency peaks at line rate • Adding color is a good first example of source-aware channel coding. • What was the problem? • What was the solution? • Why does it work? • What are its drawbacks (i.e., when does it fail)? Comp 294-9 :: Fall 2003

10. Interlacing • Progressive = every scan line done in turn. • Interlaced = every other scan line done. • Creates two fields: odd field and even field. • NTSC: 262.5 lines per field at 60 fields per sec. • PAL: 312.5 lines per field at 50 fields per sec. • Fields are separated in time. Comp 294-9 :: Fall 2003

11. Analog Bandwidths • True meaning of the bandwidth. • Y, U, and V signals are all continuous along a scan line. • A bit of a hybrid signal: discrete vertically, continuous horizontally. • NTSC: • Y is 4.2 MHz wide, I is 2 MHz, Q is 1 MHz • PAL: • Y is 6 MHz wide, U is 3 MHz, V is 3 MHz Comp 294-9 :: Fall 2003

12. Corresponding Data Rates • How much data can you put in 6 MHz band? • Depends on S/N ratio. • Depends on modulation scheme. • Typical: 27 - 36 Mbs • How many cable channels do you get? • If 50, then 1.3 - 1.8 Gbs coming into your home. • The real question for multimedia is: • Why haven’t we found a better use for 1.3 Gbs than continuous broadcast of Real World. Comp 294-9 :: Fall 2003

13. Digital Video Frames • Almost always progressive • 3 planes of pixel values (Y, U, and V) • Pixel depth • Geometry of each plane: width x height • Chrominance is generally subsampled. • How the planes relate to each other. • Frame rate. Comp 294-9 :: Fall 2003

14. CCIR-601 • Standard established for digitizing NTSC and PAL signals. Comp 294-9 :: Fall 2003

15. Why 8-bits for chrominance? • What’s another way to cut chrominance bandwidth in half? • Use 4-bits per pixel. • Why won’t that work? • Need the dynamic range for color. • But what about when Y is either small or large? • Don’t need the range, but lots of YUV combinations that won’t ever be used. Comp 294-9 :: Fall 2003

16. 4:2:2 • For every 4 luminance samples, take 2 chrominance samples from odd lines and 2 from even lines. • Chrom. planes just as tall, half as wide. • JPEG does this. Comp 294-9 :: Fall 2003

17. 4:2:0 • 2 chrominance samples for every 4 luminance samples, odd lines only. • Chrominance halved in both directions. • MPEG generally does this. Comp 294-9 :: Fall 2003

18. 4:1:1 • What should this be? • 1 chrominance for every 4 luminance for both odd and even lines. • And that is what it is. • But sometimes used to refer to this: Comp 294-9 :: Fall 2003

19. De-interlacing • Since analog video fields are separated in time by 1/2 the frame rate, at least half of a digital frame is missing no matter where you sample from. • Normal solution: linear interpolation of even fields to produce matching samples for odd fields. • Even better: linear interpolation of both fields to produce matching samples and thus digital frame rate will equal field rate. Comp 294-9 :: Fall 2003

20. Film Frame Rates • Film is a different beast altogether. • Continuous both vertically and horizontally • Projection is simultaneous for all parts of the picture. • Expensive medium. • Combination of all of this motivates 24 fps. • Film to digital is easier than video to digital. • No interlacing, sample where you want to. • Film to video is harder. Comp 294-9 :: Fall 2003

21. 1 2 3 4 5 1e 1o 2e 2o 3e 2o 3e 3o 4e 4o 5e 5o 3:2 Pulldown • Converts 24 frames to 60 fields. Comp 294-9 :: Fall 2003

22. Overall sampling lesson • Can’t recover what you don’t have. • Conversion between representations requires estimation of missing samples. • Interpolation causes errors: • Spatially: at the edges. • Temporally: when moving. Comp 294-9 :: Fall 2003

23. Common Digital Video Sizes • CCIR-601 720x480 4:2:2, 4:2:0 • SIF 360x240 4:2:0 • CIF 360x288 4:2:0 • 4:3 HDTV 1440x1152 4:2:2, 4:2:0 • 9:16 HDTV 1920x1152 4:2:2, 4:2:0 • 4CIF, 16CIF, QCIF Comp 294-9 :: Fall 2003

24. Digital Bitrates • Current television: • 30 fps * 720 * 480 * 1.5 * 8 = 124 Mb/s • 9:16 HDTV • 30 fps * 1920 * 1152 * 1.5 * 8 = 796Mb/s • This motivates compression. Comp 294-9 :: Fall 2003