NaradaBrokering Grid Messaging and Applications as Web Services

1. NaradaBrokeringGrid Messaging and Applications as Web Services Geoffrey Fox Community Grids Lab Indiana University gcf@indiana.edu

2. NaradaBrokering might Support • Grid Messaging reliably in spirit of WS-ReliableMessaging • Virtualize inter-service communication • Federate different Grids Together • Scalable pervasive audio-video conferencing • General collaborative Applications and Web services • Build next generation clients interacting with messages not method-based user interrupts • Unify peer-to-peer networks and Grids • Handle streams as in “media or sensor Grids” • Handle events as in WS-Notification

3. NaradaBrokering Audio/Video Conferencing Client Computer Modem Server Peers NaradaBrokering Broker Network Minicomputer Firewall Laptop computer Workstation Peers Audio/Video Conferencing Client PDA Web Service B Queues Stream Web Service A

4. “GridMPI” v. NaradaBrokering • In parallel computing, MPI and PVM provided “all the features one needed’ for inter-node messaging • NB aims to play same role for the Grid but the requirements and constraints are very different • NB is not MPI ported to a Grid/Globus environment • Typically MPI aiming at microsecond latency but for Grid, time scales are different • 100 millisecond quite normal network latency • 30 millisecond typical packet time sensitivity (this is one audio or video frame) but even here can buffer 10-100 frames on client (conferencing to streaming) • 1 millisecond is time for a Java server to “think” • Jitter in latency (transit time through broker) due to routing, processing (in NB) or packet loss recovery is important property • Grids need and can use software supported message functions and trade-offs between hardware and software routing different from parallel computing

5. NaradaBrokering • Based on a network of cooperating broker nodes • Cluster based architecture allows system to scale in size • Originally designed to provide uniform software multicast to support real-time collaboration linked to publish-subscribe for asynchronous systems. • Now has several core functions • Reliable order-preserving “Optimized” Message transport (based on performance measurement) in heterogeneous multi-link fashion with TCP, UDP, SSL, HTTP, and will add GridFTP • General publish-subscribe including JMS & JXTA and support for RTP-based audio/video conferencing • Distributed XML event selection using XPATH metaphor • QoS, Security profiles for sent and received messages • Interface with reliable storage for persistent events

6. Laudable Features of NaradaBrokering • Is open sourcehttp://www.naradabrokering.org • Has client “plug-in” as well as standalone brokers • Will have a discovery service to find nearest brokers • Does tunnel through most firewalls without requiring ports to be opened • Supports uniform time across a distributed network • Supports JXTA, JMS (Java Message Service) and more powerful native mode • Transit time < 1 millisecond per broker • Will have setup and broker network administration module

7. NaradaBrokering Naturally Supports • Filtering of events to support different client requirements (e.g,. PDA versus desktop, slow lines, different A/V codecs) • Virtualization of addressing, routing, interfaces • Federation and Mediation of multiple instances of Grid services as illustrated by • Composition of Gridlets into full Grids (Gridlets are single computers in P2P case) • JXTA with peer-group forming a Gridlet • Monitoring of messages for Service management and general autonomic functions • Fault tolerant data transport • Virtual Private Grid with fine-grain Security model

8. Service Consumer SOAP+HTTPGridFTPRTP …. Messaging Substrate Any Protocol satisfying QoS Grid Messaging Substrate SOAP+HTTPGridFTPRTP …. Standard client-server style communication. Consumer Service Substrate mediated communication removes transport protocol dependence. e.g. Now can use GridFTP with multi-stream performance boost for any message flow

9. Virtualizing Communication • Communication specified in terms of user goal and Quality of Service – not in choice of port number and protocol • Protocols have become overloaded e.g. MUST use UDP for A/V latency requirements but CAN’t use UDP as firewall will not support ……… • A given communication can involve multiple transport protocols and multiple destinations – the latter possibly determined dynamically NB Brokers FastLink FirewallHTTP B1 SatelliteUDP A Hand-HeldProtocol B2 Software Multicast Dial-upFilter NB Broker B3 Client Filtering

10. Performance Monitoring • Every broker incorporates a Monitoring service that monitors links originating from the node. • Every link measures and exposes a set of metrics • Average delays, jitters, loss rates, throughput. • Individual links can disable measurements for individual or the entire set of metrics. • Measurement intervals can also be varied • Monitoring Service, returns measured metrics to Performance Aggregator.

11. Architecture of Message Layer • Need to optimize not only routing of particular messages but classic publish/subscribe problem of integrating different requests with related topics (subscribe to sports/basketball/lakers and sports) • Related to Akamai, AOL … caching and Server optimization problem Hypercube ofNB Brokers (logical not physical) N≈100 for Distance Education Per edge Broker Scale with distributed Broker net? 1-> N Grid Clients

12. NaradaBrokering Communication • Applications interface to NaradaBrokering through UserChannels which NB constructs as a set of links between NB Brokers acting as “waystations” which may need to be dynamically instantiated • UserChannels have publish/subscribe semantics with XML topics • Links implement a single conventional “data” protocol. • Interface to add new transport protocols within the Framework • Administrative channel negotiates the best available communication protocol for each link • Different links can have different underlying transport implementations • Implementations in the current release include support for TCP,UDP, Multicast, SSL, RTP and HTTP. • GridFTP most interesting new protocol • Supports communication through proxies and firewalls such as iPlanet, Netscape, Apache, Microsoft ISA and Checkpoint.

13. Mean transit delay for message samples in NaradaBrokering: Different communication hops 9 hop-2 hop-3 8 hop-5 7 hop-7 6 5 Transit Delay (Milliseconds) 4 3 2 1 0 100 1000 Message Payload Size (Bytes) Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux

17. Low Rate; Small Messages NaradaBrokering and JMS (Java Message Service) (commercial JMS)

18. NaradaBrokering and JXTA Federation • Based on hybrid proxy that acts as both Rendezvous peer (JXTA routers) and NaradaBrokering end-point. • No changes to JXTA core or constraints on interactions • Change made to Rendezvous layer • Peers are not aware that they interact with a Narada-JXTA proxy or Rendezvous peer. • NB provides JXTA guaranteed long distance delivery • NB federates multiple JXTA Peer Groups

19. End-point Services inNative NaradaBrokering • Allows you to create Consumers (subscribers) of events (an event is a time stamped message where time stamp can be empty!) • Allows you to create Producers of events (publishers) • Allows you to discover brokers and initialize communications with the broker. • Services available at the client side will perform • Compression of payloads • Computation of Message digests for Integrity • Secure encryption of payload based on the specified keys • Fragmentation of large payloads into smaller packets • Redundancy service which maintains active (alternate) connections to multiple brokers.

20. Event Consumer Capabilities • Allow you to subscribe to events that conform to a certain template. • The specified subscription profile could topic-based strings, XPath queries, <tag=value> pairs or integer topics. • Event Consumers can also create Consumer constraints to specify various properties regarding the delivery of events. • Consumer constraints are different from subscriptions. • Subscriptions (or Profiles) are evaluated in a distributed fashion by the broker network, • Consumer constraints are QoS related and are managed by the QoS services running on the end-point. • Consumer constraints can specify • Reliable Delivery of events • Ordered (Publisher, causal and time ordered) delivery of events • Exactly once delivery of events • Delivery after un-compression of compressed payload • Delivery after decrypting encrypted payload

21. Event Producer Capabilities • Facilitate the generation of events in correct format (next slide) • Facilitate the publishing of events to brokers • Allow the creation of Publisher constraints which facilitate specification of properties that need to be satisfied by published events • Among the constraints that can be specified include • Method ofSecuring message payloads • Computing message digests • Compressing message payloads • Fragmenting large payloads

22. Native NaradaBrokering Event • The event comprises of • Event headers • Content Synopsis(for selection as in JMS properties WITHOUT reading body) • Content Payload • Dissemination Traces (generated on the fly as event traverses broker network) • This is different from structure of JMS or JXTA events • This NBEvent structure supports the extra capabilities discussed earlier • The event headers specify information regarding • Security and Integrity of encapsulated payload • Fragmentation of events • Compressionof payloads • Correlation identifiers (to define ordering between different streams as is needed in some collaboration applications) • Priority • Application Type • Event Identifiers

23. Based on Message Level Security • Messages organized into topics • Each topic has a separate key; Topics can be organized into sessions

26. Autonomic Services • In a Web (Grid) Service architecture, the state of any service is defined by its initial condition and all the messages (including ordering) that it receives • This how shared event model of collaboration works • This is a “Finite State Change” model analogous to saving file and “undo” command in many editors • NB plus a robust store can “guarantee” to save all these messages for (all) services • This allows one to build both "autonomic data transport" and "autonomic services" since these services can sustain packet losses in transport and can also sustain failures of apps/brokers • archived messages (previous invocations, published events etc) can be retransmitted to reconstruct state at the service or to correct a transport error. • Further anomalies in message traffic (such as a publisher or subscriber are silent) can be detected by NB and signal problems • We are building examples of both scenarios using GridFTP as our data transport example • We will build a sample autonomic visualization service with detection of failed servers and brokers

27. Reliable Messaging Standards • There are 2 competing standards in the Web Services community • WS-Reliability from OASIS • WS-ReliableMessaging from IBM and Microsoft (now expanded to include others)

28. WS-Reliability & WS-RM • Specifications provide reliable delivery between two endpoints. • Both the specifications use positive acknowledgements to ensure reliable delivery • Both specifications include support for faults • WS-Reliability is a SOAP based protocol • WS-ReliableMessaging provides an XML schema for reliable messaging. • Includes a SOAP binding.

29. NaradaBrokering & Reliable Delivery specifications • We can provide support for both these specifications • In NaradaBrokering we provide reliable delivery from multiple points to multiple points • We have identified issues that will allow federation between these specifications • Sequence numbering, fault mappings, numbering rollovers, quality of service guarantees • Federation would allow • WSRM sender & WS-Reliability receiver • WS-Reliability sender & WSRM receiver

31. NaradaBrokering, WS-Notification & JMS • NaradaBrokering is JMS compliant • Topics in NaradaBrokering could be based on XML, String(as in JMS), Plain text, Integers, and (tag=value) tuples. • Subscriptions could be XPath queries, SQL queries, Regular expressions, Strings and integers • Almost all the primitives needed in WS-Notification are available in NaradaBrokering • Exception: Entities never communicate directly with each other, as proposed in WS-Notification. • We are either allow such direct communication or mimic in NB – no performance overhead! • We are currently building a prototype implementation of WS-Notification • Need to relate WS-Notification with WS-Eventing and WS-Events

32. NaradaBrokering and NTP • NaradaBrokering includes an implementation of the Network Time Protocol (NTP) • All entities within the system use NTP to communicate with atomic time servers maintained by organizations like NIST and USNO to compute offsets • Offset is the computed difference between global time and the local time. • The offset is computed based on the time returned from multiple atomic time servers. • The NTP algorithms weighs results from individual time clocks based on the distance of the atomic server from the entity. • NTP ensures that all entities are within 1-30 milliseconds of each other. • The timestamps account for clock drifts that take place on machines • Time returned is based on software clocks which can slow down with increased computing load on the machine.

33. NB Time Service • The following results are obtained over a period of 48 hours. • In these tests, NB Time Service computes offset every 30 seconds. • Usually NTP daemons on Linux/Solaris adjust underlying clock every second. • In figure on left, there is a NTP daemon running on the local machine besides the NB Time Service. NTP daemon running on the machine adjusts the underlying system clock. NB Time Service shows consistent results with NTP daemon. • Figure on right shows results on machine that does not run NTP daemon.

34. Nugget3 Nugget4 Interaction Nugget1 Nugget2 Data Building PSE’s with theRule of the Millisecond I • Typical Web Services are used in situations with interaction delays (network transit) of 100’s of milliseconds • But basic message-based interaction architecture only incurs fraction of a millisecond delay • Thus use Web Services to build ALL PSE components • Use messages and NOT method/subroutine call or RPC • This “rule” OFTEN violated – people use “Java Interfaces” and so cannot distribute services

35. Building PSE’s with theRule of the Millisecond II • Messaging has several advantages over scripting languages • Collaboration trivial by sharing messages • Software Engineering due to greater modularity • Web Services do/will have wonderful support • “Loose” Application coupling uses workflow technologies • Find characteristic interaction time (millisecond programs; microseconds MPI and particle) and use best supported architecture at this level • Two levels: Web Service (Grid) and C/C++/C#/Fortran/Java/Python • Major difficulty in frameworks is NOT building them but rather in supporting them • IMHO only hope is to always minimize life-cycle support risks • Science is too small a field to support much! • Expect to use DIFFERENT technologies at each level even though possible to do everything with one technology • Trade off support versus performance/customization

36. Streams and Workflow • Grids need to consider streams and Services • Topics in NB are streams not just individual messages • There is service-oriented workflow where streams are typically implicit. • For stream-oriented workflow, the streams are explicit. We have built a sophisticated system GlobalMMCS for audio-video conferencing, which we will discuss next • Media Industry and sensor based science are different examples • We are building a stream control engine for NaradaBrokering when streams are “just” message flows on the Grid. Here one would use NB discovery services – find streams – and monitor • NB could be used as part of workflow runtime

37. q storage1 y2 z2 UNIX-style Workflow Example • `flow: x &> (y1|z1 &> p,(q|storage1)), (y2|z2|storage2)`; • Note this approach allows for example all workflow streams to use RMI, GridFTP, RTP – your or rather NaradaBrokering’s choice y1 z1 p x storage2 NaradaBrokering Topic (Queue)

38. Stream–oriented Workflow • As in audio-video conferencing and multimedia file delivery where it’s the media streams that are the “point” • Services generate and transform streams but one thinks of streams going through services rather than services generating streams • Multi-cast streams where video from one client sent to all other participants in a collaborative session common • One thinks of a stream being published and participants subscribing to it. Subscribe Pub/Sub Queue Publish

39. InterGrids Federated Grid using NB • Build a P2P Network where each component (cell or Gridlet)is itself a Grid • If cell is a single computer, reduces to using NB to build communication infrastructure between nodes of P2P network • If cell is a JXTA peer group, then InterGrids includes previous federation of JXTA Peer Groups NB Brokers Gridlets Grid formed from Multiple cells

40. InterGrids Mediation Architecture • NB acts as a Mediation agent in such a Cellular Grid • Using federated security model constructs a VPN like Virtual Private Grid (NB could support VPN protocol and combine with Grid Security) • Mediation includes more than routing (as in current JXTA) as can map between Interface standards • Each Gridlet can use different Service standards • Services register interfaces with mediator giving ways to map using perhaps OGSA as a common intermediate form • Allows integration of OGSI WSRF WS-GAF and “pure web service” or Jini or JXTA based Grids where each Grid uses its natural service architecture • Support interoperable (like Job Submission) and federated (like registry or metadata catalog) services • Exploits stream filtering capability of NB

41. Collaboration and Web Services • Collaboration has • Mechanism to set up members (people, devices) of a “collaborative sessions” • Shared generic tools such as text chat, white boards, audio-video conferencing • Shared applications such as Web Pages, PowerPoint, Visualization, maps, (medical) instruments …. • b) and c) are “just shared objects” where objects could be Web Services but rarely are at moment • We can port objects to Web Services and build a general approach for making Web services collaborative • a) is a “Service” which is set up in many different ways (H323 SIP JXTA are standards supported by multiple implementations) – we should make it a WS

42. Shared Event Collaboration • All collaboration is about sharing events defining state changes • Audio/Video conferencing shares events specifying in compressed form audio or video • Shared display shares events corresponding to change in pixels of a frame buffer • Instant Messengers share updates to text message streams • Microsoft events for shared PowerPoint (file replicated between clients) as in Access Grid • Finite State Change NOT Finite State Machine architecture • Using Web services allows one to expose updates of all kinds as messages • “Event service” for collaboration is similar to Grid notification service and we effectively define SDE’s (service data elements) in OGSI • Group (Session) communication service is needed for the delivery of the update events • Using Event Messaging middleware makes messaging universal

43. Collaborative SVG Web Service • SVG is W3C 2D Vector Graphics standard and is interesting for visualization and as a simple PowerPoint like application • Further SVG is built on W3C DOM and one can generalize results to all W3C DOM-based applications (“all” in future?) • Apache Batik SVG is Java and open source and so it is practical to modify it to explore • Real Applications as a Web Service • Collaboration as a Web Service • MVC model and web services with implications for portlets • We use NaradaBrokering and XGSP to control collaboration; support PDA Cell-phone and desktop clients; are restructuring Batik as MVC Web Service • Good progress in all areas see • http://www.svgarena.org for SVG Games • http://grids.ucs.indiana.edu/ptliupages/projects/carousel/ for PDA

44. Applications as Web Services? • Build “all” classic applications in Web service style with user interface and “real application” interacting by (WSDL/SOAP) messages and NOT by O/S controlled interrupts • This is “just” MVC (Model View Control) paradigm done very explicitly • Quite hard because MVC not actually used very systematically • Perhaps the advantages of this architecture could be enough to shake the Microsoft hegemony in both O/S and “productivity” applications • Current challenges of Microsoft applications are trying to do as well and not easy to see how they can do better • Immediately make a lot easier to support cross O/S applications • Form the “Next Generation Client Consortium”? • There is quite a lot of open source (but not web service based) software with which to begin

45. Classic MVC Paradigm

46. Data Resource Facing Ports Application as a Web service W3C DOM Semantic Events Model NaradaBrokering User FacingPorts Events as Messages Rendering as Messages Control W3C DOM Raw (UI) Events View W3C DOM User Interface Web Service Model for Application Development Natural inMVC Model Interrupts in traditional monolithic applications become“real messages” not directly method calls Natural for collaboration and universal access

47. Collaborative SVG As A Web Service NaradaBrokering

48. Collaborative SVG Chess Game in Batik Browser Players Observers

49. Shared Output Port Collaboration WSDL R U Application orContent source F F WSViewer WSDisplay I I O O Web Service Collaboration as a WSSet up Session with XGSP Web Service Message Interceptor Master WS Viewer WS Display Text Chat Whiteboard Multiple masters Event(Message)Service OtherParticipants WS Viewer WSDisplay

50. NaradaBrokering SVGBrowser SVGBrowser SVGBrowser SVGBrowser SVG Model (DOM) SVGViewer SVGViewer SVGViewer SVGViewer NaradaBrokering SIMD Collaboration Non Web Service Implementation Identical Programs receiving identical eventsToken determines if browser is moving, waiting for opponent or an observer Shared Output portSIMD CollaborativeWeb Service