Streaming Video Over the Internet

1. Streaming Video Over the Internet Andreas Panteli 03/05/2012 ECE 654 Advanced Computer Networks Streaming Video Over The Internet

2. Introduction • Video principles • When an image appears on the retina it is retained for some millisecond before decaying • If a sequence of images is drawn line by line at 50 images/sec the eye does not notice that it is looking at discrete images • Streaming Video • Video content need not be downloaded in full • It is being played out while parts of the content are being received and decoded • Real-time multimedia have timing constraints • Audio and video data must be played out continuously • Bandwidth, delay and loss requirements • No QoS guarantees to streaming video, from the current best-effort Internet! Streaming Video Over The Internet

3. An Architecture for Video Streaming Streaming Server Client/Receiver Media Synchronization Video Decoder Audio Decoder Storage Device Video Compression Application-layer QoS Control Application-layer QoS Control Compressed Video Raw Video Audio Compression Compressed Audio Raw Audio Transport Protocol Transport Protocol Internet (Continuous media distribution services) Streaming Video Over The Internet

4. Key Areas • Video Compression • Application-Layer QoS Control • Continuous Media Distribution Services • Media Synchronization Mechanisms • Protocols for Streaming Media Streaming Video Over The Internet

5. Key Areas • Video Compression • JPEG standard • MPEG standard • Application-Layer QoS Control • Continuous Media Distribution Services • Media Synchronization Mechanisms • Protocols for Streaming Media Streaming Video Over The Internet

6. Digital Video • Digital video is represented as a sequent of frames, each consisting of a rectangular grid of picture elements (pixels) • For color video, 8 bits for each of the RGB colors are being used • 24 bits/pixel => 16 million colors • To produce smooth motion, digital video, we must display at least 25 frames/sec, but to avoid flickering we need to repaint each frame on the screen twice • Smoothness and flickering affects the bandwidth requirements • Consider the lower video configuration for computer monitor 1024x768 with 24 bits/pixel and 25 frames/sec • Need to be fed at 472 Mbps • We need video compression! Streaming Video Over The Internet

7. Video Compression • Two algorithms are needed • Encoding and Decoding • Asymmetries of the algorithms • Acceptable for the encoding algorithm to be slow and require expensive HW, if the decoding algorithm is fast • Only for video on demand • For real-time multimedia like video conferencing, slow encoding is unacceptable • Acceptable to have a slightly different video signal than the original, after encoding and decoding (Lossysystem) • Lossy systems are important • Accepting a small amount of information loss can give a huge payoff in terms of the compression ratio possible Streaming Video Over The Internet

8. The JPEG Standard • Compression algorithm for still images • Could be applied to each image in succession to achieve video compression • Compute the luminance Y and the two chrominances I and Q • Y = 0.30R + 0.59G + 0.11B • I = 0.60R – 0.28G - 0.32B • Q = 0.21R – 0.52G + 0.31B • Compress the two chrominances twice than the luminance • Lossy reduction, but the eye barely notices it since it responds to luminance more than to chrominance • JPEG often produces 20:1 compression or better • Decoding a JPEG image requires running the algorithm backwards Streaming Video Over The Internet

9. The MPEG Standard (1/2) • Main algorithms for video compression and international standards since 1993 • Can compress both audio and video • Three parts in MPEG-1 • Audio • Video • System (integrates the other two) Streaming Video Over The Internet

10. The MPEG Standard (2/2) • Two kinds of redundancies in videos • Spatial redundancy • Coding each frame separately with JPEG • Temporal redundancy • Consecutive frames are often almost identical • MPEG-1 output consists of four kinds of frames • I (Intracoded) frames: Self-contained JPEG-encoded still pictures • P (Predictive) frames: Block-by-Block difference with the last frame • B (Bidirectional) frames: Differences between the last and the next frame • D (DC-coded) frames: Block averages used for fast-forward • MPEG-1 can achieve a compression factor of 40 • Decoding has about the same complexity as encoding Streaming Video Over The Internet

11. Key Areas • Video Compression • Application-Layer QoS Control • Congestion control • Error control • Continuous Media Distribution Services • Media Synchronization Mechanisms • Protocols for Streaming Media Streaming Video Over The Internet

12. Application-Layer QoS Control • Avoids congestion and maximizes video quality in the presence to packet loss Congestion Control Error Control FEC Delay-Constrained Retransmission Error-Resilient Encoding Error-Concealment • Rate Control • Source-Based Rate Control • Receiver-Based Rate Control • Hybrid-Rate Control • Rate Shaping • Frame-dropping filter • Layer-dropping filter • Frequency filter • Re quantization filter Streaming Video Over The Internet

13. Rate Control (1/2) • Technique to determine the sending rate of video traffic based on the estimated available bandwidth of the network • Source-based rate control • The sender is responsible for adapting the video transmission rate based on feedback from the receivers • Probe-based approach • Sending rate adjustment to maintain the packet loss ratio below a certain threshold • Model-based approach Streaming Video Over The Internet

14. Rate Control (2/2) • Receiver-based rate control • The receivers regulate the receiving rate of video streams by adding/dropping channels • Probe-based approach • When no congestion is detected, join a channel • => increase of it’s receiving rate • When congestion is detected, drop a channel • => reduction of it’s receiving rate • Model-based approach • Explicit estimation for the available network bandwidth • Also based on Equation (1) • Hybrid rate Control • Combination of the above mentioned techniques Streaming Video Over The Internet

15. Rate Shaping • Matches the rate of a pre-compressed video bit stream to the target rate constraint • A rate shaper (or filter) is required for source based rate control • Types of filters • Codec filter • Performs transcoding between different compression schemes • Frame-dropping filter • I, P ,B , D frames in MPEG • Frequency filter • Operates in the frequency domain (DCT coefficients) • Re-quantization filter • Re-quantizes the DCT coefficients with a larger quantization step Streaming Video Over The Internet

16. Error Control (1/2) • Forward Error Correction (FEC) • Add redundant information so that original message can be reconstructed in the presence of packet loss • Channel coding • Stream is chopped into segments • Each segment is packetized into k packets • A block code generates an n-packet block, where n>k • User only needs to receive any k packets out of n of the block • Source coding • N-th group of blocks contains redundant information of (n-1)th group of blocks • Joint source/channel coding • Optimal rate allocation between Channel and Source coding Streaming Video Over The Internet

17. Error Control (2/2) • Delay-constraint Retransmission When the receiver detects the loss of packet N if (Tc + RTT + Ds < Td(N)) send the request for packet N to the sender Where Tc current time RTT estimated round-trip time Dsa slack term (e.g receiver’s decoding delay) Td(N) time when packet N is scheduled for display • Error-resilient Encoding • Enhances robustness of compressed video to packet loss • A video is compressed into multiple streams (descriptions) • Each description provides acceptable visual quality • Combined descriptions provide a better visual quality • Error Concealment • Performed by the receiver when packet loss has already occurred • Spatial interpolation • Missing pixel values are reconstructed using neighboring pixels • Temporal interpolation • Loss data is reconstructed using data from the previous frame Streaming Video Over The Internet

18. Key Areas • Video Compression • Application-Layer QoS Control • Continuous Media Distribution Services • Network Filtering • Application-level Multicast • Content Replication • Media Synchronization Mechanisms • Protocols for Streaming Media Streaming Video Over The Internet

19. Network Filtering • Adequate support from the network is critical in order to provide quality multimedia presentations • Routers have no knowledge of the format of the media streams • Randomly discard packets • Filters receive the client’s requests and adapt the stream sent by the server accordingly • Typically, frame-dropping filters are used • Drop packets in a way that gracefully degrades the stream’s quality • Increase bandwidth efficiency by discarding late frames R: Router Streaming Video Over The Internet

20. Application-level Multicast • Internet’s original design fails to effectively support streaming-media multicast • IP multicast has barriers • Scalability, network management, support for higher layer functionality (e.g. error, flow and congestion control) • Application-level multicast to the rescue • Builds a multicast service on top of the Internet • Media bridge • Routing in the application layer • Each Media Bridge is interconnected with one or more neighboring Media-Bridges through explicit configuration • Collectively employ a distributed application-level multicast routing algorthm Unicast Multicast Streaming Video Over The Internet

21. Content Replication • Advantages • Increased network’s bandwidth efficiency • Reduced load on streaming servers • Reduced latency for clients • Increased availability Mirroring Caching Exploits temporal locality Cache retrieves data from server Clients retrieve data from cache if available Cache sharing and cache hierarchies • Scatter copies of the original multimedia files in different locations around the Internet • Clients retrieve data from nearest duplicate server Streaming Video Over The Internet

22. Key Areas • Video Compression • Application-Layer QoS Control • Continuous Media Distribution Services • Media Synchronization Mechanisms • Intra-stream Synchronization • Inter-stream Synchronization • Inter-object Synchronization • Protocols for Streaming Media Streaming Video Over The Internet

23. Media Synchronization • Multimedia applications • Various integrated media streams that must be presented in a synchronized fashion • Media streams may lose synchronization after moving from the server to the client • Three layers of synchronization • Intra-stream synchronization • Maintain continuity of logical data units • Inter-stream synchronization • Maintain temporal relationships among different continuous media • e.g. movements of the lips of a speaker do not correspond to the presented audio • Inter-object synchronization • Relation with time-independent data, such as text, still image and streams • e.g. during a presentation, audio is commenting one slide while another slide is presented Streaming Video Over The Internet

24. Media Synchronization Mechanisms • Delay in the network is unpredictable due to the best-effort nature of the Internet • Axes-based specifications, or time-stamping • At the source, a stream is time-stamped to keep temporal information within the stream and with respect to other streams • Mechanisms implemented on the end systems • Preventive mechanisms • Minimize latencies and jitters • i.e. disk-reading scheduling algorithms, network transport protocols, operating systems and synchronization schedulers • Corrective mechanisms • Recover synchronization in the presence of synchronization errors • i.e. Stream synchronization protocol (SSP) • Concept of ”intentional delay” Streaming Video Over The Internet

25. Key Areas • Video Compression • Application-Layer QoS Control • Continuous Media Distribution Services • Media Synchronization Mechanisms • Protocols for Streaming Media • Transport Protocols • Session Control Protocols Streaming Video Over The Internet

26. Protocols for Streaming Video • Data Plane • Compressed video/audio is packetized at the real-time transport protocol (RTP) layer • RTP packetized streams are then passed to the UDP/TCP layer and IP layer • IP packets are transported over the Internet • Control Plane • Real-time control protocol (RTCP) and real-time streaming protocol (RTSP) packets are multiplexed at the UDP/TCP layer • IP packets are transported over the Internet Streaming Video Over The Internet

27. Transport Protocols • UDP and TCP provide basic transport functions • UDP and TCP • Multiplexing, Error control (e.g. checksum) • TCP only: • Retransmission, Congestion control, Flow control • UDP is typically employed as the transport protocol for video streams • No packet delivery guarrantees • Receiver needs to rely in upper layer to detect packet loss • RTP and RTCP run on top of UDP/TCP • RTP : does not guarantee QoS or reliable delivery • Time stamping • Sequence numbering • Payload type identification • Source identification • RTCP : designed to work in conjunction with RTP • QoS feedback • Participant identification • Control packets scaling • Inter-media synchronization • Minimal session control information Streaming Video Over The Internet

28. Session Control Protocols • RTSP (Real-Time Streaming Protocol) • Support VCR-like control operations • (stop, pause/resume, fast forward, fast backward) • Choosing delivery channel (UDT, multicast UDT, TCP) • Establish and control streams of continuous audio and video media between the media servers and the clients • Media retrieval: ask server to setup a session to send the requested media data • Adding media to an existing session: server or client can notify each other about any additional media becoming available to the established session • SIP (Session Initiation Protocol) • Similar to RTSP • Support user mobility by proxying and redirecting request Streaming Video Over The Internet

29. References • A. S. Tanenbaum, Computer Networks, 4th ed. New Jersey: Prentice Hall, 2002. • DapengWu, Hou, Y.T., WenwuZhu, Ya-Qin Zhang, Peha, J.M., "Streaming video over the Internet: approaches and directions“,Circuits and Systems for Video Technology, IEEE Transactions on , vol.11, no.3, pp.282-300, Mar 2001 • DapengWu, YiweiThomsHou,Ya-Qin Zhang, "Transporting real-time video over the Internet: challenges and approaches," Proceedings of the IEEE , vol.88, no.12, pp.1855-1877, Dec 2000 • DapengWu and Y. Thomas Hou and Jason Yao and Y. Thomas and Hou Jason Yao and H. Jonathan Chao, “Real-time Video over the Internet: A Big Picture”, IEEE NetWorld+Interop, 2000 Streaming Video Over The Internet

30. Questions? Thank you! Streaming Video Over The Internet