Introduction to Middleware

1. Introduction to Middleware The protocols used for conference media streams The protocols to setup and control various aspects of a conference, such as membership, session information, media activity, floor control and so on.

2. Introduction to Middleware Here middleware is the glue software try to combine the multimedia system together, normally it includes Transport Protocols Session Directories, Advertisement and Invitation Protocols; Conference Control Specifications.

3. Transport Protocols • Difference between TCP and UDP at the viewpoint of multimedia • Multimedia transport protocols vs. conventional data services • A basic framing service is needed • Multiplexing may be needed • Timely delivery and trade off against reliability • Synchronization

4. Transport Protocols • RTP, the preferred carrier/framing protocol for multimedia traffic, is carried on top of UDP. It has the potential for further multiplexing information, since it carries source identifiers. • RTCP, the Real Time Control Protocol, is used to convey additional information such as participant details and statistics of packet arrivals and loss.

5. Roadmap • The transport protocol for multimedia provides some functions that are different than those for elastic reliable data transfer • RTP provides payload identification, multiplexing, feedback from receivers to senders of reception conditions and hooks for synchronization. This chapter will also talk about transportation reliability of multimedia.

6. TCP Adaption Algorithms • TCP will smoothly degrade performance of everyone as load increases(rather than blocking access). This is done by adaption algorithms. • Adaption in TCP is both to RTT and send rate. • Based on the idea of adaption, multimedia protocols adapt to delay at the receiver to: • Alter adaptive playout buffer to smooth playback • Synchronize streams from different sources at a receiver

7. TCP Adaption Algorithms • Algorithm named Exponential Weighted Moving Average: • Assume we measure the arrival time for each (i'th) packet as IATi, then the simple average would be: Av(IATi) = [ sumi=1 to n ( IATi) ]/n. But since the average is not fixed, the most recent measurements are more confidential, so use: • Av(IATi) = IATi*alpha +(1- alpha)*Av(IATi-1), where 1> alpha > 0.5 • Once have a mean IAT, required playback buffer can be calculated( roughly twice of the variation of IAT)

8. TCP Adaption Algorithms • Synchronization between two sites by exchange of packets • send a packet from s to d with source time in t1, its arrives at d when clock of d reads t2 • send the packet back to s with t1, t2 and t3(the time at d when sending) • s gets the packet at the time when its clock is at t4 • if s and d have clocks that differ by offset, and network delay is delay • t4=t3+offset+delay, t2=t1-offset+delay • so: delay=(t4+t2-t3-t1)/2, offset=(t4+t1-t2-t3)/2

9. Playout Algorithms • For a variety of reasons, samples may arrive at different times than they were sent. The variation of arrival time is a second order effect compared with the delivery delay. This can be coped with playout buffer with the size of the variation of the mean of the arrival time. • Audio, video and other interactive application receivers generally use an adaptive playout scheme. • Senders are generally adaptive to reported network conditions. The report should be constrained to be a constant percentage of the total session.

10. MPEG Systems • MPEG systems is the control part of the MPEG standard. It addresses the combining of one or more streams of video and audio as well as other data into a single or multiple streams which are suitable for storage or transmission. The stream is Packetised Elementary Stream(PES).

11. Transport and Program Streams • Transport Stream(TS), can carry multiple programs simultaneously, and is optimized for use in applications where data loss may be likely. • Program Stream(PS), is optimized for multimedia applications, for performing systems processing in software and for MPEG-1 compatibility.

12. Transport and Program Streams • Synchronization • The basic principle of MPEG system coding is the use of time stamps provided by RTP, which specify the decoding and display time of audio/video and the time of reception of the multiplexed coded data at the decoder. • This method allows a great deal of flexibility in such areas as decoder design, the number of streams, multiplex packet lengths, video/audio frame rates digital storage medium or network performance.

13. RTP • RTP is the Internet Standard for conveying media streams between interactive participants. • RTP Packet Format • RTP Header Compression • RTP Multiplexing • RTCP Packet Format • Payloads • RTCP Scaling Properties and Timer Considerations

14. RTP Packet Format • 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • |V=2|P|X| CC |M| PT | sequence number | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | timestamp | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | synchronization source(SSRC) identifier | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | contributing source(CSRC) identifiers | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • The first twelve octets are present in every RTP packet, while the list of CSRC identifiers is present only when inserted by a mixer.

15. RTP Header Compression • The combination of the IP,UDP and RTP control information adds up to a significant overhead for small media samples. IP(20 bytes)+UDP(8 bytes)+RTP(12 bytes)=40 bytes • A technique for reducing the overhead of packets, designed for compressing TCP/IP headers specified in RFC 1144 by Jacobson, was adapted by Casner and Jacobson to RTP headers, shows a typical reduction to a header size of 3-4 bytes for all control overhead.

16. RTP Header Compression • The technique consists two parts: • Noting fields in the packet headers that do not change over the life of a flow. These are the second order information which can stored in the compression state • Noting that there are few flows at the edge of the network, so such information can be conveyed over the first hop by a single packet, and subsequently referred to by a short “connection identifier, which serves to index the full state so that the first op router can reconstruct the full packet.

17. RTP Header Compression • The features of the header compression: • The compression for RTP header is basically the same as TCP/IP except in exceptional circumstances • The compression state is a soft state. It can be recovered if lost since the packet conveys enough implicit information that end-to-end checksums are still computed, and hop-wise recomputed from the state vector and from the remaining data in the compressed header. That is to say, when the router resets/changes or the network alters states, the invalid checksum causes a reset of the compressed state so that a new packet is created again.

18. RTP Multiplexing • There are a number of circumstances that multiple media flows within a single RTP data payload between two points is needed. There are at least two ways to multiplex data: • Assuming all samples have the same payload types, so just use a mapping table to save offsets for each end-to-end flow. • A more generic method suggested by Mark Handley is to adapt the ideas from previous section about RTP header compression and to allow for multiple compressed headers within a single RTP packet, one for each of the samples.

19. RTCP Packet Format • 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • |V=2|P| RC | PT=SR=200 | length | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | SSRC of sender | • +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ • | NTP timestamp, most significant Word (sender info ) | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | NTP timestap, lest significant word (sender info ) | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | RTP timestamp | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | sender’s packet count | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | sender’s octet count | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

20. RTCP Packet Format • +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ • | SSRC_1 (SSRC of first source ) (report block 1 ) | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | fraction lost | cumulative number of packets lost | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | extended highest sequence number received | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | interarrival jitter | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | last SR(LSR) | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | sender’s packet count | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ • | delay since last SR(DLSR) | • +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ • | SSRC_2 (SSRC of second source) ... (report block 2 ) | • +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

21. Payloads • There are quite a few payloads defined for RTP, ranging from owens designed to carry video/audio formerly available on hardware CODECs only, such as MPEG, H.261, and the various CCITT/ITU voice coding schemes, through to some more generic payloads designed to carry redundantly coded audio such as that used by INRIA’s Freephone.

22. RTCP Scaling Properties & Timer Considerations • The intention and original design goal of RTCP messages was to act as a distributed source of lightweight session data that would allow a range of highly fault tolertant, and seasonable scale mechanisms to be built including: • Membership RTCP reports carry the identifier of participants. • Loss statistics RTCP reports convey information about packet losses.

23. RTCP Scaling Properties & Timer Considerations • RTCP reports are designed to be sent periodically, with a frequency inversely proportional to the number of members. Though this was set to constrain the bandwidth consumed by RTCP to be a small fixed fraction of the total capacity of the session, there are three circumstances where this approach creates a problem: • A flood of initial packets of reports at the start • The flood of “BYE” packet at the end of session • The reports arrives at all participants

24. Synchronization • Some kinds of synchronization: • Intra-stream Synchronization inside a stream, for media like video/audio, whiteboard inside a stream • Inter-stream To see two streams’ reactions to the occurrence of a third stream • inter-media Just lip-synch

25. Intra-stream Synch • Intra-stream synchronization is a base part of the H.261 and MPEG coding systems. H.221 and MPEG systems specify an encapsulation of multiple streams as well as carrying timing information in the stream. • In the Internet, the RTP media specific timestamp provides a general way of carrying out the same above function.

26. Inter-stream Synch • The easiest way of synchronizing between streams at different sites is based on providing a globally synchronized clock. • Use a single clock which is propagated around the network • Use a clock synchronization protocol, such as NTP(Network Time Protocol), DTS(Digital Time Service). The network continually exchanges messages about clock offsets and network delays. • Or use timestamps to adjust clocks and measure delays accordingly, and then insert a baseline delay into the adaptive playout algorithms.

27. Inter-media Synch • Use options like Inter-stream Synch approaches • More generally, encapsulate the media in the same transmission stream. It is effective but may cause expensive computation at the receive side to decode all media at the same time. • Alternatively, use much the same scheme as to synchronize different sources from different places. This approach uses timestamps, assuming the media are timestamped accurately at the “real” source points which the receiver can use to synchronize multiple media.

28. Reliable Multicast Transport • “Reliable Multicast” is not people normally think to be a single protocol at a single ‘layer’ of a protocol stack. The differences are possibly: • Fate sharing • Performance • Semantics • In this section we will also talk about congestion avoidance for reliable multicast applications and reliable multicast framework.

29. Fate Sharing • Fate sharing in unicast applications means that if two applications has set up connection, then TCP can keep the connection as long as they like. However, if either party fails, the connection certainly fails. • Fate sharing between multicast endpoints is a more subtle idea. It is a application-oriented question. Normally we must permit any recipient timeouts/Fails.

30. Performance • The performance measurement is different for a talks to b, from a talks to b,c,d,e,f. • The previous situation is limited by 1 path, what can be done to improve the throughput(or delay bound) is done by IP. • For the second situation, the throughput or delay may not be that sustainable even in the slowest transferring state.

31. Semantics • Applications such as software distribution are cited as classic 1-n service, telemetry is given as a n-1 reliable protocol, shared whiteboards are cited as examples of n-m. • 1-n and n-1 services are those where reliability can be dealt with by rounding up the missing bits at the end of the transfer. • N-m processes need timely recovery. For each recipient, the timely order is very important.

32. Congestion Avoidance for Reliable Multicast • One of the big research areas not quite complete is how to achieve flow and congestion control for reliable multicast applications. • There are a lot options when confronted with a congested link and sending to a group. The best option depends on the nature of the application. The options can be viewed as sending at the slowest,average or fastest rate. • The global congestion picture is not at all clear for either reliable or unreliable multicast.

33. Reliable Multicast Framework • Handley proposed a framework about reliable multicast protocol. He described the categories of parameters of the system: • Number of source: 1-to-n, n-to-m or other. • Transmitting start time: Is there such a time? Is there one for each receiver marked at join time? • Real time-Consideration: About the latency constraints on delivery of data. • Consistency. • He listed congestion control and ordering as non-factors.

34. Reliable Multicast Framework • He grouped applications into five main areas: • Bulk Transfer(1-n,relaxed real time, synchronized start time, object level consistency) • Live data feeds(1 source, no start time, quasi real time, may need synchronized consistency) • Resilient(1 source, no start time, nearly real time, non consistency) • Share applications(distributed data set, relaxed consistency, many changing sources) • Hybrid - some applications maybe a hybrid of all of the above. • This will be discussed further in chapter eight.