1 / 20

MSG Status update

MSG Status update. Daniel Rodrigues. Presentation Summary. MSG overview Timeline ActiveMQ Msg -publish-simple Msg-consumer2oracle Current deployments OSG Gridview Next steps. MSG overview. MSG overview. Features of an Messaging System: Flexible architecture:

arty
Download Presentation

MSG Status update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MSG Status update Daniel Rodrigues

  2. Presentation Summary • MSG overview • Timeline • ActiveMQ • Msg-publish-simple • Msg-consumer2oracle • Current deployments • OSG • Gridview • Next steps

  3. MSG overview

  4. MSG overview • Features of an Messaging System: • Flexible architecture: • Deliver messages, either in point to point (queue)… • … or multicast mode (topics) • Support Synchronous or Asynchronous communication. • Reliable delivery of messages: • Provide reliability to the senders if required • Configurable persistency / Master-Slave. • Highly Scalable: • Network of Brokers • ActiveMQ is an Open Source message broker providing these and many other features.

  5. MSG overview • A real life working example:

  6. Timeline • 1st Trimester (Oct/Dez): • Testing of ActiveMQ 4.1, using netcat/bash scripts to test throughput; • Integrated testing with twisted stomp, python framework; • 2nd Trimester (Jan/Mar): • Simple publishers and python consumers developed by Piotr Nyczyk on place for OSG; • Msg-publish-simple and msg-consume2oracle (bugfixes + improvements + testing); • Moving to ActiveMQ 5.0, retesting throughput; • Definition of message-classes for gridview-gridftp parsing records; • Testing different broker configurations: Master-Slave, JDBC persistence, file persistence; • Testing of JMS Java clients (producers and consumers) together with python. • 3rd Trimester(Apr/): • Network of Brokers testing; • GWT (Google Web Toolkit) Client prototype. • Setting up publishers/consumers for gridview-gridftp • First 4 machines, now extended for additional 27 machines; • Definition of message-classes for downtimes with OSG and initial protyping. • Moving to ActiveMQ 5.1

  7. ActiveMQ updates • Move from 4.1 => 5.0 • Producer Flow Control! • AMQ Message Store, faster persistence • Command Agent • (Blob Messages) • Now moving from 5.0 => 5.1 • More stable • Default AMQ Message Store + bugfixes • Issues with recovering from failure: very long time to replay all kept logs! • No viable Master/Slave with fast persistence otherwise

  8. MSG publish simple • Simple python publisher script, no other dependency besides… python! • Publishes messages over either stomp or http, depending on network availability. • Currently version 0.9.3-1: • A few bug fixes since initial version • Priority to stomp over http. • Configurable logging

  9. MSG consume2oracle • Python consumer script • again no other dependency besides… python (and cx_oracle)! • How does it work: • Creates a durable subscription to the topics • Receives the messages • Retrieves the records sent along with a message • inserts them into an oracle view • Currently version 0.9.1-1: • Bug fixes • Configurable logging • Snapshot: • Implementing Status summaries topic (publishes summary information back into MSG)

  10. Current Deployments • OSG: • Broker at gridmsg001 • activemq 5.0.0 • Persistence with journaledJDBC (derby) • msgConsume2Oracle at gridmsg001 • Publishing to SAM metricOutput/metricDescription • msgPublishSimple • Several publishing from indiana.edu. • Gridview: • Broker at gridmsg002 • activemq 5.0.0 • Persistence with journaledJDBC (derby) • msgPublishSimple • Called from gridview-publisher • Deployed on 27 gridftp nodes for cms t1transfers. • msgConsume2Oracle • gridmsg002: publishing to testDatabase • Gridview machine publishing to gridview DB.

  11. Current Deployments • Testing : • Brokers: • lxb6118 / 6117 -> 5.1 Snapshot • Many different configurations along the time (Master/Slave Network of Brokers, file persistence, etc) • Publishers: • Java JMS producer (pcitgd24) • lxplus based msg-publish-simple • Consumers: • msg-consume2oracle (lxb6118/6117) • GWT test (pcitgd24)

  12. Next Steps • OSG scheduled downtimes • Message Class defined • Some test messages into a test DB done • Waiting Arvind’s prototype (very soon) • JobDetails for Dirac3 detailed logging • Interested in getting information such as cputime, and considering using MSG to do it. • Message class under definition • MSG System Management: • How to operate a messaging system in production? • We will need to know what’s going on!

  13. Thank you for your attention.

  14. Support Slides

  15. Some recent test results

  16. Some recent test results

  17. Some recent test results

  18. Some recent test results

  19. Console, GridMSG001 Broker restarts: 10 Mar. 18 Apr, too many open connections -> increase of file descriptors. Since 18 April, 220000 messages.

  20. Console, GridMSG002 420087 messages forwarded to each consumer (test + GV).

More Related