Global Multimedia Collaboration System

1. Global Multimedia Collaboration System Wenjun Wu Indiana University Bloomington IN 47401 wewu@indiana.edu http://www.globalmmcs.org

2. Outline • Service-Oriented Collaboration • Current main stream real-time collaboration technologies • Videoconferencing: H.323, SIP, Access Grid • Instant messaging & VoIP: MSN/Aol/Yahoo, Jabber, Skype • Global-MMCS introduction

3. Service Oriented Collaboration • Collaboration has • Mechanism to set up members (people, devices) of a “collaborative sessions” • Shared generic tools such as text chat, white boards, audio-video conferencing • Shared applications such as Web Pages, PowerPoint, Visualization, maps, (medical) instruments …. • b) and c) are “just shared objects” where objects could be regarded as Web Services • We can port objects to Web Services and build a general approach for making Web services collaborative • a) is a “Session Service” which is set up in many different ways

4. Shared Event in Collaboration • All collaboration is about sharing events defining state changes • Audio/Video conferencing shares events specifying in compressed form audio or video • Shared display shares events corresponding to change in pixels of a frame buffer • Instant Messengers share updates to text message streams • Microsoft events for shared PowerPoint (file replicated between clients) as in Access Grid • Using Web services allows one to expose update events of all kinds as message streams • Need publish/subscribe approach to share messages

5. Session Service in Collaboration • Membership: Participant list • Role & floor assignment • Management policy based on shared objects ( audio/video, text, whiteboard, game • Ad-hoc or formally schedule • Session is also a “shared meta object” associated with shared objects • It needs eventing to keep consistent state as well

6. Multimedia Streaming Service • Streaming-In  Stream-Out filter process media “events” between a stream source and stream sink; and can be shared • composite media service is a DAG ( directed acyclic graph) common case ~ a filter chain • QoS(in)  QoS(out) ( bandwidth, delay, jitter,loss) for example: video/audio mixing, transcoding, video-audio synchronization

7. VoIP & Videoconferencing technology

8. H.323 Introduction • Major audio-video standard but broader • “Binary” format for both “data” and “control” • Supported by many commercial vendors and used throughout the world in commercial and educational markets • Supports small-scale multipoint conferences • Has conference management functionality and the call signaling functionality H.225 ~ call set-up H.245 ~ call control H.243 ~ Audio/Video multipoint control T.120 ~ Data Collaboration

9. H.323 Protocols • H.323 is a “framework” document that describes how the various pieces fit together • H.225.0 defines the call signaling and communication between endpoints (Call Signaling) and the Gatekeeper (RAS) • Annex G/H.225.0 defines communication between Border Elements • H.245/H.243 is the conference control protocol T.120 is the data conference protocol

10. H.323 Multimedia Applications, User Interface Data Applications Media Control Terminal Control and Management H.225.0 Call Signaling H.245 V.150 T.120 T.38 RTCP H.225.0 RAS Audio Codecs G.711 G.723.1 G.729 .. Video Codecs H.261 H.263 H.264 .. RTP UDP TCP TCP/UDP UDP TCP/UDP TCP UDP IP Typical H.323 Stack http://www.packetizer.com

11. Gatekeeper (security, QoS, routing etc.) MC MP MCU Packet Switch Network .... H.323 Terminal 2 H.323 Terminal N H.323 Terminal 1 H.323 Architecture

12. SIP • Initially SIP was designed to solve problems for IP telephony. • SIP basic functions • user location resolution, • capability negotiation • call management. equivalent to the service H.225 and point to point part of H.245 • The major difference from H.323 • SIP was designed in a text format and took request-response protocol style like HTTP • SIP doesn’t define the conference control procedures like multipoint parts of H.245 and T.120.

13. sipc SIPUA Netmeeting sipconf e*phone e*phone Quick-time A Integrated SIP Service System: CINEMA From Columbia University Desktop SIP clients H.323 sip323 Gatekeeper SIP-H.323 signaling gateway sipd SIP Programmable SIP servers Conferencing Hardware SIP phone sipum Unified messaging sipgw RTSP SIP-MGCP gateway rtspd MGCP Streaming media SIP-PSTN gateway PSTN Regular telephones CINEMA clouds

14. Sipconf : SIP based Centralized conferencing sipc SIP323 sipconf Netmeeting e*phone SIP/PSTN • SIP based conferencing server • SIP/SDP and RTP/RTCP • Audio mixing • Play-out delay algorithm • Web based conference setup • G.711 A and Mu law, G.721, DVI ADPCM • Multiple simultaneous conferences http://www.cs.columbia.edu/~kns10/software/sipconf

15. Summary of H.323/SIP Conferencing Systems • Most products are Centralized conferencing system MCU integrates the service of media processing service and session management • Call-based A conference call represents control connections between clients and MCUs. MCU is just a endpoint connected to VoIP softswitch cloud • Most vendors offer hardware solutions • Thought as services and controllers but specialized protocols and implementations; NOT Service-Oriented Architectures! • Only support small size or medium size meetings ( < 20 sites )

16. Access Grid Access Grid : a large scale audio/videoconference based on a multicast network • provides the group-to-group collaborations among 150 nodes connected to Internet 2 world wide. • Use improved MBONE audiovisual tools VIC and RAT • Depends upon high-speed network ( each node needs 20Mbps ) • Peer to peer architecture for distribution with centralized non standard session control (venue server) • Did not develop many new capabilities but made existing public domain software better packaged and easier to use

17. Access Grid II • Supports multiple screens and dominates some research communities

18. Instant Messaging & VoIP

19. Proprietary IM • MSN/Yahoo/AOL • Ah-hoc small-group collaboration • Text, audio-video, gaming and others…. • Remote Presence Service ( typical publish/subscribe messaging application ) • Massive “chat servers” running behind to support millions of users across the world • Limited size of buddy list and multi-party meeting • Poor/fair quality for audio/video communication • Close protocol which is unfriendly to third-party developers

20. Skype • Skype: p2p IM&VoIP solution gained a big success. • improving sound quality ( use new iLBC audio codec ) from Global Sound • Uses a variant of IETF Stun to identify NAT and firewall • using p2p overlay (Kazaa) rather than expensive, centralized infrastructure. • provided supplemental features like instant messaging service. • Free on-net VoIP service and a fee-based off-net SkypeOut service that allows calling to PSTN and cellular phones • Millions of download and on-line users in the world

21. Why is Skype so successful? • Better voice quality excellent audio codec, fancy echo cancellation algorithm Global IP Sound ( iLBC audio codec ) • Ability to work behind firewalls and NAT • Ease of use ( quite simple UI ) based on IM metaphor • P2P style without centralized MCU any peer that has enough resource can be selected to host the mixing service limited the number of participants in a conference ( at most 4 which is common for private social meetings ) use p2p overlay to discover resources and route packets

22. But they are simply not good enough! Although all of these systems have advantages, they are not sufficient for building more advanced and integrated collaboration systems: • SIP : had a huge development recently, especially in wireless world very limited supported for conference control • H.323 : AV collaboration and T.120 are not well integrated. the AV communication services and T.120 overlay networks don’t have very good scalability. H.323 and T.120 are designed in a relative complicated OSI model. It is not easy to understand and develop in their APIs Most H.323 and SIP conferencing products are based on centralized MCU And no way to take full use of private MCU resources ( Imagine how to use these private MCUs to create a meeting that have thousands of participants )

23. But they are simply not good enough! • Access Grid • heavily depends on multicast service and limited number of uni-cast bridge servers in the Internet 2 • No way to be deployed in current Internet • Skype :Most promising • use its own propriety protocols and can’t interoperate with other legacy VoIP clients such as H.323 and SIP • only support small-scale audio conferencing ( at most 5-party ) and have no video service • Skype-2 is said to be able to support 10-party in dual-core Intel machines • Above all, no system can support medium / large size meetings in current Internet and adapt different client devices

24. What’s the ideal videoconferencing system I • A unified, scalable, robust “overlay” network is needed to support AV and data group communication over heterogeneous networking environments • go through firewall and NAT • provide group communication service in whatever unicast and multicast networks • offer reliable data delivery in whatever loss network • to be configured as P2P or distributed server-based overlay to provide differential services for VIP and regular users • A service-oriented architecture for hosting media processing service and session control service • More scalable than centralized MCU • Support various style of conferencing ( massive scale of broadcasting as well as medium size of private social meetings ) • Service providers can be highly distributed and p2p ~ Skype p2p audio mixing • Scalable service discovery based on p2p search • Customized media filters for different clients ( PC, PDA, … )

25. What’s the ideal videoconferencing system II • A core conference control mechanism is required for establishing and managing the multi-point conference • Complete conference control service like T.124 (Generic Conference Control) in T.120 framework • more flexible facilities to describe application sessions and entities ( role-based, XML ) for all kinds of collaboration: audio/video, game, whiteboard • Session border management Integrate different AV sessions ( H.323 , SIP, Access Grid, RealStreaming … ) Simply regard these bridging gateways as “add-on services”

26. Global-MMCS Service Architecture

27. Session Server XGSP-based Control Media Servers Filters NaradaBrokering All Messaging Admire SIP H323 Access Grid Native XGSP XGSP Web Service MCU Architecture Use Multiple Media servers to scale to many codecs and many versions of audio/video mixing WebServices High Performance (RTP)and XML/SOAP and .. NB Scales asdistributed Gateways convert to uniform XGSP Messaging NaradaBrokering

28. Break up into “Services” • Monolithic MCU becomes many different “Simple Services” • Session Control • Thumbnail “image” grabber • Audio Mixer • Video Mixer • Codec Conversion • Helix Real Streaming • PDA Conversion • H323/SIP Session/Signaling Gateways • As independent can replicate particular services as needed • Codec conversion might require 20 services for 20 streams spread over 5 machines • 1000 simultaneous users could require: • 1 session controller, 1 audio mixer, 10 video mixers, 20 codec converters, 2 PDA converters and 20 NaradaBrokers • Support with a stream optimized Grid Farm in the sky • Future billion way “Video over IP” serving 3G Phones and home media centers/TV’s could require a lot of computing

29. GlobalMMCS and NaradaBrokering • All communication – both control and “binary” codecs are handled by NaradaBrokering • Control uses SOAP and codecs use RTP transport • Each stream is regarded as a “topic” for NB • Each RTP packet from this stream is regarded as an “event” for this topic • Can use replay and persistency support in NB to support archiving and late clients • Can build customized stream management to administer replay, and who gets what stream in what codec • NaradaBrokering supports unicast and multicast • Use firewall penetration and network monitoring services in NB to improve Q0S

30. NaradaBrokering Queues Stream NB supports messages and streams NB role for Grid is Similar to MPI role for MPP

31. Incorporating Support for Audio/Video Delivery into NaradaBrokering • Added support for an unreliable transport protocol, UDP • Implemented a fixed size (fast) topic (8 bytes). • Designed a new compact event with minimum headers. • Added support for legacy RTP clients (both unicast clients and multicast groups) • Improved the routing algorithm to handle real-time audio and video stream delivery.

32. XGSP Conference Control Architecture

33. XML based General Session Protocol The XGSP conference control includes three services: • Conference management supports user sign-in, user create/terminate/join/leave/invite-into XGSP conferences conference calendar service • Application session management provides users with the service for creating/terminating application sessions, managing session related services such as audio/video mixing • Floor control manages the access to shared collaboration resources in different application sessions for example, in a large scale of meetings having thousands of people, only limited people are allowed to become presenters so that they can send audio/video

34. Global-MMCS Community Grid • This includes an open source protocol independent Web Service “MCU” which will scale to an arbitrary number of users and provides support for thousands of simultaneous users of collaboration services. • The function of A/V media server is distributed using NaradaBrokering architecture. • Media Servers mix and convert A/V streams • Open XGSP MCU based on the following open source projects • openh323 is basis of H323 Gateway • NIST SIP stack is basis of SIP Gateway • NaradaBrokering is open source messaging • Java Media Framework basis of Media Servers • Helix Communityhttp://www.helixcommunity.org for Real Media • http://www.globalmmcs.org open source release

35. Audio/Video Meeting Tests for single broker Audio Meeting Tests Video Meeting Tests

36. Distributed Brokers Tests • Going through multiple brokers does not introduce considerable overhead. • Scalability of the system can be increased almost linearly by adding new brokers

38. Analysis of the broker network’s performance • Test results showed that the broker network can scale well for both single large size meetings and multiple smaller size meetings. • In large size meetings, the capacity of the broker network is increased with respect to the capacity of the added brokers. • In multiple smaller size meetings, the distribution of users among brokers are important. Inter-broker stream exchange can reduce the scalability. Few users should not be scattered around the broker network. • In wide area networks, this videoconferencing system provides many benefits with distributed broker architecture: bandwidth savings, latency savings, and better quality services. • In summary, thousands of concurrent users can easily be supported in distributed broker settings.

39. media services computation overhead

40. Improved JMF Performance Fraction of CPU used versus number of received streams The CIF-size video sequence from a 30-second movie with a lot of motions is streamed to the clients. Each stream is encoded in H.261, and has average bandwidth of 400Kbps and 20 fps.

41. Global-mmcs user interface

43. Polycom, Access Grid and RealVideo views of video-mixed streams using GlobalMMCS

44. MPEG-4 vs. H.261 • We added MPEG4 video to Java Media Framework • Higher quality and flexible video sizes including distributed pixels

45. Coupled Diverse Streams • GlobalMMCS supports many diverse streams managed by “video system” • Different audio and video codecs • Shared display using video codecs (MPEG4 or H261) • Motion JPEG – stream of images to and from PDA • NaradaBrokering represents these and other collaborative streams just a “topics”; collaboration from multiple clients subscribing to a topic • Text Chat • Traditional lossless codec based shared display • White boards • Control streams • Streams can be linked to provide composite topics • eSports project linking video streams and real time annotation of any frame • Can rewind and choose any frame of a real-time stream

46. eSports Snapshot Master Video Annotation Whiteboard Collaborative Video Annotation Whiteboard Synchronized Replay of archived video and annotation

47. Esports with sport movies

48. Integration of PDA, Cell phone and Desktop Grid Access NB Support for optimizedPDA Communication

49. PDA Download video as images

50. GlobalMMCS Status/Futures I • 1. New Collaboration tools • Shared IDL (Visualization), PowerPoint, OpenOffice (Applications need a month or so more) • SVG game ( stable ) • Whiteboard ( stable ) • e-Sport ( prototype) • Jabber IM client ( prototype) • XGSP needs extension to support • 2. JMF Audio/Video client ( stable) • performance enhancement finished • new codec ( MPEG4-DivX finished; try MPEG4-Xvid and H.264) • support different platform ( Linux, Mac – Mac well developed but need to chase bug(s) ) • support NAT/firewall transparently like Skype • Google Desktop PlugIn (under development )