Multimedia

1. Multimedia Outline Compression RTP Scheduling

2. Compression Overview • Encoding and Compression • Huffman codes • Lossless • data received = data sent • used for executables, text files, numeric data • Lossy • data received does not != data sent • used for images, video, audio

3. Lossless Algorithms • Run Length Encoding (RLE) • example: AAABBCDDDD encoding as 3A2B1C4D • good for scanned text (8-to-1 compression ratio, used for fax) • can increase size for data with variation (e.g., some images) • Differential Pulse Code Modulation (DPCM) • example AAABBCDDDD encoding as A0001123333 • diff. can be encoded by fewer bits than original symbol • change reference symbol if delta becomes too large • works better than RLE for many digital images (1.5-to-1)

4. Dictionary-Based Methods • Build dictionary of common terms • variable length strings • Transmit index into dictionary for each term • Lempel-Ziv (LZ) is the best-known example • Commonly achieve 2-to-1 ration on text • Variation of LZ used to compress GIF images • first reduce 24-bit color to 8-bit color (most closely approximate) • treat common sequence of pixels as terms in dictionary • not uncommon to achieve 10-to-1 compression

5. Image Compression • JPEG: Joint Photographic Expert Group (ISO/ITU) • Lossy still-image compression • Three phase process • process in 8x8 block chunks (macro-block) • DCT: • transforms signal from spatial domain to the frequency domain (loss-less) • higher freq. finer detail (multi-resolution analysis) • apply a quantization to the results (lossy) • RLE-like encoding (loss-less) • Color image • each pixel is three values: YUV (luminance + 2 chrominance), more suitable for human visual system than RGB • Compression ratio 30-to-1

6. Quantization and Encoding For the coefficient at 0,0 Quantum = 3 If the value is 25 Quantized value [25/3]=8 Decompressed value 8*3=24 • Quantization Table 3 5 7 9 11 13 15 17 5 7 9 11 13 15 17 19 7 9 11 13 15 17 19 21 9 11 13 15 17 19 21 23 11 13 15 17 19 21 23 25 13 15 17 19 21 23 25 27 15 17 19 21 23 25 27 29 17 19 21 23 25 27 29 31 • Encoding Pattern Use RLE for 0’s, Huffman code for others

7. MPEG • Motion Picture Expert Group • Lossy compression of video • First approximation: JPEG on each frame • Also remove inter-frame redundancy

8. MPEG (cont) • Frame types • I frames: intrapicture • P frames: predicted picture • B frames: bidirectional predicted picture • Example sequence transmitted as I P B B I B B

9. MPEG (cont) • B and P frames • coordinate for the macroblock in the frame • motion vector relative to previous reference frame (B, P) • motion vector relative to subsequent reference frame (B) • delta for each pixel in the macro block • Effectiveness • typically 90-to-1 • as high as 150-to-1 • 30-to-1 for I frames • P and B frames get another 3 to 5x

10. Other Video Encoding Standards • The version previous described is MPEG-2 • MPEG-4 • Support for lower speed links such as wireless • Treat a scene as a collection of video objects rather than macro-blocks as in MPEG-2 • H.261, H.262 • Standardized by ITU-T • Similar compression methods as MPEG • Used for lower speed links (ISDN, 64 kbps increments) • Compare to MPEG, 1.5 Mbps

11. Audio Compression: MP3 • CD Quality • 44.1 kHz sampling rate • 2 x 44.1 x 1000 x 16 = 1.41 Mbps • stereo, 16 bits per channel, higher quality than dig. Tel. (64kbps) • 49/16 x 1.41 Mbps = 4.32 Mbps • Overhead ratio 49/16, for synchronization, error correction • Layering for transmission on low speed links • Layer I, 384 kbps, comp. ratio 4 • Layer II, 192 kbps, comp. ratio 8 • Layer III, 128 kbps, comp. ratio 12 (MP3)

12. MP3 Encoding • Strategy • similar to MPEG • split into some number of frequency bands • divide each subband into a sequence of blocks • encode each block using DCT + Quantization + Huffman • trick: how many bits assigned to each subband • Determined by human listening sensitivity and experience

13. The Real-Time Transport Protocol • RTP is a transport protocol for multimedia applications but implemented in the application space. • RTP multiplexes several real-time streams into a single stream of UDP packets • A companion protocol, Real-Time Transport Control Protocol (RTCP), provides control functions (RTP is for data transport) • Provide feedback (delay, jitter, bandwidth, congestion, etc.), synchronization, text info about source, etc.

14. RTP • RTP has no flow control, error control, acknowledgement and retransmission; • RTP has sequence numbers • the receiver will not ask for retransmission but will interpolate instead. • RTP provides timestamps • relative to the start of the stream • enable buffering and correct playback at the receiver, and synchronizing among constituent streams (indicated by the synchronizing source identifier in the header). • RTP defines application profiles • i.e., MP3, indicated by the payload type field in the header.

15. RTP header

16. Transmitting MPEG • Adapt the encoding to network bandwidth • Resolution, frame rate, quantization table, GOP (group of picture, I, B, P frames) mix • Packetization • Application layer framing, packetizing at, e.g., macroblock boundaries (to isolate loss to one block) • Dealing with loss • B frame loss damages a single frame, I frame loss multiple frames • Retransmission not an option, can adapt Diffserv dropping probability • GOP-induced latency • B frame delayed until the next I/P frame (n frames away) • @15 fps, can be n*67ms, perceivable to human • Many video conference application use Motion-JPEG, whose frames are standalone.

17. Layered Video • Layered encodeing • e.g., wavelet encoded • Receiver Layered Multicast (RLM) • transmit each layer to a different group address • receivers subscribe to the groups they can “afford” • Probe to learn if you can afford next higher group/layer • Smart Packet Dropper (multicast or unicast) • select layers to send/drop based on observed congestion • observe directly or use RTP feedback

18. Real-Time Scheduling • Priority • Earliest Deadline First (EDF) • Rate Monotonic (RM) • Proportional Share • with feedback • with adjustments for deadlines