Concepts of Multimedia Processing and Transmission

1. Concepts of Multimedia Processing and Transmission IT 481, Lecture #8 Dennis McCaughey, Ph.D. 30 October, 2006

2. Broadcast Environment IT 481, Fall 2006

3. Video Transmission System Example IT 481, Fall 2006

4. Rate Control • Video applications involve real-time display of the decoded sequence. • Transmission across a Constant Bit Rate (CBR) channel requires a constant end-to-end delay between the time that the encoder processes a frame and the time at which that same frame is available to the decoder • A buffer is require to match the Variable Bit Rate (VBR) encoder with the CBR channel IT 481, Fall 2006

5. Admission Control • Decision if a new connection with a given set of QoS parameters can be allowed into the network • Criteria: the new connection will not degrade the QoS of other ongoing connections • Need models for predicting the expected bit rates of video sources especially packet switched networks • Simpler for circuit switched networks where the transmission resources are constant over the duration of the connection IT 481, Fall 2006

6. Usage Parameter Control • Prevent sources from exceeding the traffic parameters negotiated at call setup • Maliciously or unintentionally • Example: tracking cell rate in an ATM network and verifies that the source remains within that rate • Example: “Leaky Bucket” • a cell counter incremented with each cell arrival • Decremented at fixed intervals • If counter exceeds a threshold cells are dropped or marked for deletion IT 481, Fall 2006

7. Multi-Resolution Encoding • Separation of the information into two or more layers or resolutions • Coarse Resolution • Reduce the spatial or temporal resolution of the sequence or by having images of lower quality • Transmitted by high priority packets • Detailed Resolution • Transmitted by lower priority packets than can be discarded first • Multi-Resolution Encoding enables efficient error concealment techniques • e.g. interpolate lost information from packets that were not lost IT 481, Fall 2006

8. Layered Video Coding • Provide multiple image quality levels simultaneously across multiple network channels • e.g. Standard Resolution and High Definition TV • Each receiver individually tunes its reception rate by adjusting the number of layers that it receives • Two problems to solve • Layered Compression • Multiple image quality levels • Layered Transmission Problem • Selective delivery of layer subsets to individual receivers IT 481, Fall 2006

9. Structure of a Simulcast Coder IT 481, Fall 2006

10. Simulcast Coder • Produces a multirate set of signals that are independent of one another • Each layer provides improved quality, independent of sublayers • Single layer (nonscalable) decoder can decode any layer • Total available bandwidth is partitioned dependent on the desired quality for each independent sublayer IT 481, Fall 2006

11. Structure of a Layered Coder IT 481, Fall 2006

12. Layered Compression • Input signal is compressed into a number of discrete layers, arranged in a hierarchy that provides progressive refinement • If first layer is received, decoder will produce the lowest quality version of the signal • If the decoder receives two layers, it will combine the second layer with the first layer to produce improved quality • Overall, the quality improves with the number of layers that are received and decoded • Layered compression and layered transmission must be combined to be effective Slide courtesy of Hung Nguyuen IT 481, Fall 2006

13. Simulcast vs. Layered Approach • Distortion measures quality degradation between reconstructed and original signals • DI(R): an ideal coder • DR(R): a real coder • DL(R): a layered coder • DS(R): a simulcast coder IT 481, Fall 2006

14. Layered Transmission • Each layer is transmitted on a different network channel. • Network only forwards the number of layers that each physical link can support. • Each user receives the best quality signal that the network can deliver. • The network must be able to drop layers selectively at each bottleneck link IT 481, Fall 2006

15. Video Communication System IT 481, Fall 2006

16. Error Resilience • Redundancy is added to the compressed bitstream to allow the detection and correction of errors • Can be added in either the source or channel encoder • Shannon Information Theory: • Separately design the source and channel coders to achieve error-free transmission so long as the source is represented by a rate below the channel capacity • Source should compress the source as much as possible • Channel coder, via Forward Error Correction (FEC) adds redundancy bits to enable error detection and correction IT 481, Fall 2006

17. Binary Symmetric Channel IT 481, Fall 2006

18. Shannon’s Capacity Theorem • If the Rate (R) of a code R = log2(m)/L is less than channel capacity C, there exists a combination of source and channel encoders such that the source can be communicated over the channel with fidelity arbitrarily close to perfect • m = Number of message words • L = Number of code word bits IT 481, Fall 2006

19. Types of Codes • Block Codes • Hamming Codes • Bose-Chaudhuri-Hocquenhem BCH Codes • Reed-Solomon Codes • Convolutional Codes IT 481, Fall 2006

20. Example: (n,k) Hamming Code (n,k) = (2m-1,2m-1-m), for any m IT 481, Fall 2006

21. (7,4) Hamming Code Corrects all single errors IT 481, Fall 2006

22. Parity Check Matrix IT 481, Fall 2006

23. Decoding IT 481, Fall 2006

24. (7,4) Messages and Codes(Systematic Code) 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 0 0 1 0 1 0 1 1 0 0 1 1 1 1 0 0 0 1 0 0 1 1 0 1 0 1 0 1 1 1 1 0 0 1 1 0 1 1 1 1 0 1 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 1 1 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 0 1 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 0 1 1 1 0 0 1 0 0 1 1 0 1 0 1 0 0 1 0 1 1 1 0 1 1 1 0 0 0 0 0 1 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 Messages Code Words IT 481, Fall 2006

25. Received Code Words 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 0 0 0 1 0 0 1 0 1 0 1 1 0 1 1 0 0 0 0 1 1 0 0 1 1 1 1 0 0 0 1 1 1 1 0 1 0 1 1 1 1 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 1 0 0 1 0 1 0 0 1 1 1 1 1 0 1 1 1 1 0 0 0 0 1 1 0 0 1 1 0 1 1 1 0 1 0 1 1 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 1 1 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 0 1 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 0 1 1 1 0 0 1 0 0 1 1 0 1 0 1 0 0 1 0 1 1 1 0 1 1 1 0 0 0 0 0 1 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 errors Received Words Transmitted Words IT 481, Fall 2006

26. Syndromes 1 0 0 0 1 0 0 0 1 1 1 0 0 1 1 1 1 1 1 0 1 1 0 0 0 1 0 0 0 1 1 1 0 0 1 1 1 1 1 1 0 1 1 0 0 0 1 0 4 2 1 6 3 7 5 4 2 1 6 3 7 5 4 2 Syndrome Table 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 • Rows in Syndrome Table indicate error locations • Syndromes point to rows Binary Decimal IT 481, Fall 2006

27. Performance • Bit Error Probability p=10-6 • M = 100 Bits • Probability that message is received correctly, • Pc=(1-10-2)100 = .367 • Use a hamming (7,4) code • 25 7-Bit code words with 4 info bits per word • Probability that a word is correctly decoded • Pw = probability that one or fewer bits are decoded incorrectly • =(1-10-2) + 7 (10-2) =.998 • Probability that message is received correctly • =(.998)25 = .95 • A substantial improvement IT 481, Fall 2006

28. Video Streaming Architecture IT 481, Fall 2006

29. Scalable and Non-Scalable Coder/Decoder Scalable Encoder/Decoder Scalable Encoder/Decoder • A nonscalable video encoder generates one compressed bit-stream. • Scalable video encoder compresses a raw video sequence into multiple sub-streams IT 481, Fall 2006

30. Video Stream Architecture Building Blocks • Video Compression • Application Layer QoS Control • Continuous Media Distribution Services • Streaming Servers • Media Synchronization • Protocols for Multimedia IT 481, Fall 2006

31. Video Compression • Video compression schemes can be classified into two categories: scalable and nonscalable • Scalable video is capable of gracefully coping with the bandwidth fluc­tuations in the Internet. • Primarily concerned with scalable video coding techniques. We will also discuss the requirements imposed by streaming applications on the video encoder and decoder. IT 481, Fall 2006

32. Application Layer QoS Control • Congestion control takes the form of rate control. • Three kinds of rate control: • 1) Source-based, • The source-based rate control is suitable for unicast; • Most recent studies on source-based rate control have been focused on TCP-friendly adaptation • A number of TCP-friendly adaptation schemes have been proposed and demonstrated to achieve certain degree of fairness among competing connections, including TCP connections. • However, strictly TCP-like rate control may result in sharp reductions in the transmission rate, and possibly unpleasant visual quality [66]. • Needs further investigation on how to trade off responsiveness in detecting and reacting to congestion with smooth fluctuation in visual quality. • 2) Receiver-based, and 3)Hybrid rate-control. • Both are suitable for multicast since both can achieve good tradeoff between bandwidth efficiency and service flexibility for multicast video. IT 481, Fall 2006

33. Source Based Rate Control IT 481, Fall 2006

34. Continuous Media Distribution Services • Continuous media distribution services are built on top of the best-effort Internet with the aim of achieving QoS and efficiency for streaming video. • A major topic of active research is how to build a scalable, efficient, cost-effective and incremental deployable infrastructure for continuous media distribution. IT 481, Fall 2006

35. Streaming Servers • Essential in providing streaming services. • Current research efforts include: • 1) Efficient support VCR-like interac­tive control; • 2) The design of efficient and reliable storage and retrieval of multimedia objects on disk arrays; • 3) The design of highly scalable multimedia servers in a variety of environments ranging from video-on-demand servers to integrated multimedia file system • 4) The design of fault-tolerant storage systems • Desirable features of both parity and mirroring • Trade off the parity group size with the number of disks across which original data of a single disk is replicated for mirroring. IT 481, Fall 2006

36. Media Synchronization • Intra-stream synchronization: The lowest layer of con­tinuous media or time-dependent data (such as video and audio) is the media layer. • Maintains the continuity of logical data units. • Prevents pauses and gaps. • Inter-stream synchronization: The second layer of time-dependent data is the stream layer. • Maintains temporal relationships among different continuous media. • Prevents skew between the streams may become intolerable. • eg, users could be annoyed if they notice that the movements of the lips of a speaker do not correspond to the presented audio. • Inter-object synchronization: The highest layer of a mul­timedia document is the object layer, which integrates streams and time-independent data such as text and still images. • Synchronization at this layer is referred to as inter-object synchronization. • The objective of inter-ob­ject synchronization is to start and stop the presentation of the time-independent data within a tolerable time interval, • e.g., the audience of a slide show could be annoyed if the audio is commenting one slide while another slide is being presented IT 481, Fall 2006

37. Protocols for Multimedia • Network Layer Protocol • Transport Protocol • Session Control Protocol • UDP • TCP • RTP • RTCP IT 481, Fall 2006

38. Standardized Protocols • Several protocols have been standardized for communication between clients and streaming servers. • Future research topics on design of protocols include: • 1) How to take caches into account (e.g., how to communicate with continuous media caches and how to control continuous media caches); • 2) How to efficiently support pause/resume operations in caches (since the pause/resume operations interfere with the sharing of a multimedia stream among different viewers); and • 3) How to provide security in the protocols IT 481, Fall 2006

39. Unicast and Multicast Multicast video distribution using point-to-multipoint transmission. Unicast video distribution using multiple point-to-point connections. IT 481, Fall 2006

40. Streaming Server Components • Communicator: • A communicator involves the application layer and transport protocols implemented on the server. • Through a communicator, the clients can communicate with a server and retrieve multimedia contents in a continuous and synchronous manner. • Operating system: • Different from traditional operating systems, • An operating system for streaming services needs to satisfy real-time requirements for streaming applica­tions. • Storage system: • A storage system for streaming services IT 481, Fall 2006

41. Protocol Stack for Multimedia IT 481, Fall 2006

42. RTP • RTP does not guarantee QoS or reliable delivery, but rather, provides the following functions in support of media streaming: • Time-stamping: RTP provides time-stamping to synchronize different media streams. • Sequence numbering: RTP employs sequence numbering to place the incoming RTP packets in the correct order. Since packets arriving at the receiver may be out of sequence (UDP does not deliver packets in sequence),. • Payload type identification: The type of the payload contained in an RTP packet is indicated by an RTP-header field called payload type identifier. • The receiver interprets the content of the packet based on the payload type iden­tifier. • Certain common payload types such as MPEG-audio and video have been assigned payload type numbers • For other payloads, this assignment can be done with session control protocols. • Source identification: The source of each RTP packet is identified by an RTP-header field called Synchronization Source identifier (SSRC), which provides a means for the receiver to distinguish different sources. IT 481, Fall 2006

43. RTCP • QoS feedback: This is the primary function of RTCP. • RTCP provides feedback to an application regarding the quality of data distribution. • The feedback is in the form of sender reports (sent by the source) and receiver re-ports (sent by the receiver). • The reports can contain in-formation on the quality of reception such as: • 1) Fraction of the lost RTP packets, since the last report; • 2) Cumulative number of lost packets, since the beginning of re­ception; • 3) Packet interarrival jitter; and • 4) Delay since receiving the last sender’s report. • Participant identification: A source can be identified by the SSRC field in the RTP header. • RTCP provides a human-friendly mechanism for source identification. • RTCP SDES (source description) packets contain textual information called canonical names as globally unique identifiers of the session participants. • It may include a user’s name, telephone number, email address, and other information. IT 481, Fall 2006

44. RTCP • Control packets scaling: To scale the RTCP control packet transmission with the number of participants, a control mechanism is designed as follows. • The control mechanism keeps the total control packets to 5% of the total session bandwidth. • Among the control packets, 25% are allocated to the sender reports and 75% to the receiver reports. • To prevent control packet starvation, at least one control packet is sent within 5 s at the sender or receiver. • Inter-media synchronization: RTCP sender reports contain an indication of real time and the corresponding RTP timestamp. This can be used in inter-media synchroniza­tion like lip synchronization in video. • Minimal session control information. This optional functionality can be used for transporting session information such as names of the participants. IT 481, Fall 2006

45. Session Control Protocols • RTSP functions • Support VCR-like control operations such as stop, pause/re­sume, fast forward, and fast backward. • Provides a means for choosing delivery channels (e.g., UDP, mul­ticast UDP, or TCP), and delivery mechanisms based upon RTP. • RTSP works for multicast as well as unicast. • Also establishes control streams of continuous audio and video media between the media servers and the clients. • Specifically, RTSP provides the following operations. • Media retrieval: The client can request a presentation description, and ask the server to setup a session to send the requested media data; • Adding media to an existing session: The server or the client can notify each other about any additional media becoming available to the established session IT 481, Fall 2006

46. Session Control Protocols • Similar to RTSP, SIP can also create and terminate sessions with one or more par­ticipants. • Unlike RTSP, SIP supports user mobility by proxying and redirecting requests to the user’s current location IT 481, Fall 2006

47. References • D. Wu et. Al. “Streaming Video over the Internet: Approaches and Directions”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 11, No.3, March 2001 IT 481, Fall 2006