Messaging Systems for the Grid

1. Messaging Systems for the Grid Daniel Rodrigues

2. Summary • Messaging Systems Overview • Monitoring context in the Grid • The MSG – Messaging System for Grids • Fast Forward

3. Messaging Systems • Before going any further, the philosophy: “Software development trend is to somehow mimic real world!” – Daniel Rodrigues • Procedural Programming  Beaureaucracy • Object Oriented  World entities and interaction • Aspects  Cut through the mess! • Agents  Real People. • Messaging Systems  Communication • It might be sound, image, snailmail, etc.

4. Messaging Systems • Why use messaging? • For communicating we could use: • File transfer • Shared Databases • Remote Procedure Invocation • Web Services • Mail • CORBA • They do exist; • They have common ideas; • They share implementations; • You might be using more than one to achieve a result that suits your needs! “Now look, you know different people think about life in different ways. Lawyers think life is a big court room; Doctors probably thinks life is like a big operation; Bus drivers think life is...er...a big bus I guess. Who knows what the hell those guys think. Anyway, I've always thought of life as a big football game...” Black Grape, England’s Irie

5. Messaging Systems • Why use messaging? • Key ideas and benefits: • Loosely coupled distributed communication; • Exceptional interoperability; • Asynchronous; • Reliable; • Configurable Persistence (just like your tax collector) • Drawbacks: • More complex programming model (we do like bureaucracy after all  ) • Harder to do sequenced and synchronous model • Performance? (maybe FTP could do the trick)

6. Messaging Systems • Ok, may we finally see a picture? Publisher Publisher Publisher Publisher Consumer

7. Messaging Systems • That’s all? • Enterprise Integration Patterns • Designing, Building and deploying Messaging Solutions • Gregor Hohpe / Bobby Woolf • Core Patterns • Some not so wild Patterns

8. Messaging Systems • Patterns: Message • Header • Routing information • Description • Body • Data • Ignored by the messaging system • EventMessage, CommandMessage, DocumentMessage, RequestReply • Could be SOAP, JMS, Stomp, etc.

9. Messaging Systems • Patterns: Message Channel • Point-to-Point • Snail Mail • Queues • Publish-Subscribe • Television/radio Broadcast • Topics • DataTypeChannel, InvalidMessageChannel, DeadLetterChannel, ChannelAdapter, MessageBus, MessagingBridge

10. Messaging Systems • Patterns: Message Endpoint • Publisher • Gets data from application and creates a message. • Consumer • Extracts data from a message and passes it on to the application. • SelectiveConsumer, CompetingConsumer, DurableSubscriber, MessageDispatcher, TransactionalClients, EventDrivenConsumer. • Endpoints either sends or receives messages, and are channel specific. (Ears mouth,eyes are not the same thing)

11. Messaging Systems • Other Patterns • Message Routers • Message may be routed to different channels depending on its characteristics; • Simple Example: use a wild card topic! • grid.usage.transfer.*, where it will be forwarded to grid.usage.transfer.<INFRASTRUCTURE> • MessageTranslators • Translation at different layers (data structure, types, representation, or transport). • e.g. transport protocols: TCP => HTTP => SOAP => JMS • Pipes and Filters • Message may need processing in different steps. • A Message goes through filtering and pipes that perform different functions (e.g, authN, authZ)

12. Messaging Systems • Isn’t it too complex to implement all this? • Indeed. • But someone has already done most of the work for you: • Commercial solutions: • Tibco Rendezvous, IBM WebSphere MQ, SUN Java Message Service, Microsoft MSMQ, BEA MessageQ, SonicMQ, 29West UME/LBM. • OpenSource providers: • Apache ActiveMQ, ObjectWeb JORAM, Open JMS. • Each are adequate to different problems. • Integration on different platforms; • Latency concerns; • High throughputs;

13. Messaging Systems • Where is it used? • Financial Services • exchanges, brokerages, hedge funds; • Insurance Companies • Banking Industry • Telecoms • Usually embedded in integrated solutions • Enterprise Backbones; • WebsphereMQ example (March 2007): • 10.000 customers • 10 billion messages carrying US$1 quadrillion (US$ 1 000 000 000 000 000) worth of business transactions.

14. Summary • Messaging Systems Overview • Monitoring context in the Grid • The MSG – Messaging System for Grids • Fast forward

15. Monitoring Context • How does Message Oriented Middleware fit into the WLCG monitoring context? • Grid is a complex infrastructure, with many different services deployed in different environments. • We need to monitor the services in order to: • Know when an action to repair is necessary; • Help improve the overall reliability; • Provide stakeholders with current and historical status information. • A vast amount of monitoring data is produced • Local fabric monitoring( e.g., Nagios, LEMON) • Remote monitoring (e.g., SAM)

16. Monitoring Context • Who is involved (stakeholders)? • Site Administrators • Grid Operators • CIC on Duty • Regional Operation center • WLCG Project management • Virtual Organizations • WLCG Experiments • Monitoring developers + operators

17. Monitoring Context • High Level Model: LEMON GridView Experiment Dashboard R-GMA Nagios GOCDB GridView HTTP GridIce Dashboard LDAP SAM GridIce SAME GridView

18. Monitoring Context • WLCG Monitoring Working Group: • Initially focused on stakeholder requirements • Distill into a set of architectural principles • Propose some new technologies to help • Reuse of standard commodity components • Used to design site-local monitoring prototype • An attempt to extend this to a more global view • Knowing that operations model is changing from central to regional/national/local • Looking on the architectural principles…

19. Monitoring Context • Reduce time to respond • “Site administrators are closest to the problems, and need to know about them first” • Tell others what you want to know • “If you’re monitoring a site remotely, it’s only polite to give the data to the site” Chris Brew • Remote systems should feed back information to sites • Don’t impose systems on sites • Cannot dictate a monitoring system

20. Monitoring Context • No monolithic systems • Different systems should specialize in their areas of expertise • No central bottlenecks • “Local problems detected locally shouldn’t require remote services to work out what the problem is” • Specific Visualization for each stakeholder • All are using same underlying data

21. Monitoring Context • The starting point is what we have now: • Availability testing framework – SAM/RSV • Job and Data reliability monitoring –Gridview • Grid topology – GOCDB/Registration DB • Dynamic view of the grid – BDII/CeMon • Accounting – APEL/Gratia • Experiment views – Dashboards • Fabric monitoring – Nagios, LEMON, … • Grid operations tools – CIC Portal • They work together right now • To a certain extent !

22. Monitoring Context

23. Monitoring Context • We need: • Loose coupling of systems • Distributed components • Reliable delivery of messages • Standard methods of communication • Flexibility to add new producers and consumers of the information without having to reconfigure everything • Message Oriented Middleware provides this • And is widely used in similar scenarios

24. Monitoring context • Reliablity and persistence of messaging built into the broker network. • Mitigates the single point of failures we’ve had with previous solutions

25. Monitoring context • Not a silver bullet • Still can end up with spaghetti • Tight specification of interaction of components • Message format specifications • Standard metadata schema • Message Queue naming schemas • Protocols • System management is key • You’ve got code for free from the messaging system • But you need to write your management layer • Component co-ordination • Configuration • Message tracing • Debugging

26. Monitoring context • Conclusion • The monitoring context is highly distributed; • Many components could benefit from gathering common information in a reliable, flexible way; • MOM is a way of leveraging the current underlying infrastructures;

27. Monitoring Context • A real life working example:

28. Monitoring Context Transparent Broker Network Messaging System Adaptor Database archiver component Standard components Standard process Application WLCG Monitoring – some worked examples - 28

29. Summary • Messaging Systems Overview • Monitoring context in the Grid • The MSG – Messaging System for Grids • Fast forward

30. MSG Overview • An infrastructure providing an easy way to send messages; • Each message has a well defined format adhering to a message class specification • Well defined set of message classes • Three main components: • Apache ActiveMQ broker; • msg-publish-simple; • msg-consume2oracle; • Using file-based SAN persistency; • Publish-Subscribe Channels (Topics) • Durable Subscribers

31. MSG: Message • Message endpoints on a topic should: • Consumers: expect a well formatted message • Producers: send a properly formatted message • Message Classes: • To each corresponds a specification • One message may contain multiple records • Each record consists of plain text key-value pairs, terminated by “EOT” • A few fields are mandatory: Consumers are expecting them! • Some fields may be sent as an header (for later filtering using selectors)

32. MSG: Message • Example: transferProtocol: GridFTP publishingHost: lxfsrc5807.cern.ch voName: cms srcHost: lxfsrc5807.cern.ch destHost: c2fs008.grid.sinica.edu.tw gridftpStreams: 10 numberBytes: 2684354560 fileName: //castor/cern.ch/cms/store/PhEDEx_LoadTest07_4/LoadTest07_CERN_3e6 startTime: 20-05-2008T13:17:07.514952Z endTime: 20-05-2008T13:33:58.156241Z userName: cms001 EOT transferProtocol: GridFTP publishingHost: lxfsrc5807.cern.ch voName: cms srcHost: lxfsrc5807.cern.ch destHost: diskserv-san-20.cr.cnaf.infn.it gridftpStreams: 3 numberBytes: 2684354560 fileName: //castor/cern.ch/cms/store/PhEDEx_LoadTest07_4/LoadTest07_CERN_F1 startTime: 20-05-2008T13:17:46.811483Z endTime: 20-05-2008T13:34:21.227585Z userName: cms001 EOT destination: /topic/grid.usage.transfer persistent:true transferProtocol: GridFTP msgEncodedTime: 2008-05-21T22:29:57,712Z

33. MSG: Apache ActiveMQ • Powerful OpenSourceMessageBroker • Currently running v4.1 & v5.1 • Message Channels • Publish-Subscribe; • Point to Point; • VirtualDestinations, Wildcards, CompositeDestinations; • Synchronous / Asynchronous sending. • Wide range of supported protocols and clients • Open Wire for high performance clients; • STOMP (Simple Text Oriented Protocol); • REST, XMMP, AMQP;

34. MSG: Apache ActiveMQ • Configurable persistence • JDBC + High performance journal • File based MessageStore (Since 5.0) • Clustering • Master/Slave failover • Provides High Availability • Network of Brokers • Avoid Client/server || hub/spoke single point of failure • Store and forward with consumer priority • Increasing Scalability • Consumers and Producers load balancing • Selectors • Discovery

35. MSG: msg-publish-simple • Send messages into the Message Channel • Validates well formatted against message class; • Reassembles records according to selected headers; • Very lightweight script • Depends only on Python > 2.3 • Uses python asyncore • Designed to run anywhere (e.g. WN’s) • Can use many broker endpoints (will select one which is available) • Use either STOMP or plain HTTP

36. MSG: msg-consume2oracle • Consumes messages • Creates a durable subscription; • Can read different message classes on different topics (one durable subscription per topic!) • Publishes into Oracle. • Extracts records from incoming messages; • Inserts records into an Oracle View, corresponding to the message class definition. • Only need to worry about the trigger! • Configurable system management • Publishes back client status information • Messages received in a topic; • Records inserted of a given message class; • Very lightweight script • Depends only on Python > 2.3 • also cx_oracle

37. MSG: performance • Extensive testing of broker many features under different configurations • Test results available on twiki, here are some: • Broker ran for 6 weeks with no crashes • 50 million messages of several sizes (0 to 10 kB) forwarded to consumers; • 12 million incoming messages from producers; • Up to 40 producers/80 consumers; • Stable under irregular testing pattern; • Setting persistence limits throughput.

38. MSG: performance • Throughput testing

39. MSG: performance • Testing persistency

40. MSG: performance • Testing persistency

41. MSG: performance • Testing clustering • Fast internal openwire!

42. MSG: results • Flagship: OSG RSV – SAM bridge • Running since January. • Crashed once, because there were not enough file descriptors configured.  • Gridview - GridFTP transfers. • Currently publishing from 27 cms t1transfer machines; • In testbed right now, a validation consumer;

43. Summary • Messaging Systems Overview • Monitoring context in the Grid • The MSG – Messaging System for Grids • Fast Forward • In the monitoring context.

44. MSG: results • Migrating to Regions

45. MSG: results • Messaging based archiving & reporting

46. Thank you for your attention. Additional Questions?

47. Thank you for your attention. Additional Questions?