1 / 15

DIRAC Information System

DIRAC Information System. Ian Stokes-Rees. Summary. Service and Agent Configuration Motivation Architecture Example API Service and Agent Monitoring XMPP Asynchronous Messaging Protocol Architecture Messaging Types Current Usage Future Plans. Configuration System: Motivation.

guinevere-campbell
Download Presentation

DIRAC Information System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DIRAC Information System Ian Stokes-Rees

  2. Summary • Service and Agent Configuration • Motivation • Architecture • Example • API • Service and Agent Monitoring • XMPP Asynchronous Messaging Protocol • Architecture • Messaging Types • Current Usage • Future Plans

  3. Configuration System: Motivation • Configuring a large number of distributed services and agents was a complex problem • Required a mechanism that provided • Local configuration • Global configuration • Configuration sharing • Overriding of global settings • Ease of deployment • Ease of updates • Robustness • Simplicity

  4. Service Topology INI INI INI Information Service XML-RPC INI INI Service B Service C Service A Service D Agent X Agent Y

  5. Example: INI Format Section [lbnts2] Modules= JobAgent [Agent] LogFile= agent.log LogOutput= stdout,file LogLevel= DEBUG [JobAgent] CEUniqueIds= in2p3.fr/pbs-short AgentName= TestModularAgent Option Value

  6. Features • Creates image of a single INI file by “merging” local and remote configuration • Warnings are produced for discrepencies between local and remote settings • NOTE: Another Service may expose it’s own details via the remote InformationService API • Service (or Agent) may add, change, or remove a remote Information Service at any time • Currently only supports one remote Information Service • In principal, nothing wrong with ordered list of Information Services to cycle through

  7. API • Simple, but powerful • Based on core Python ConfigParser module • Roughly follows Windows INI format • Base methods • value = get(section,option) • void set(section,option,value) • list = options(section) • list = sections()

  8. XML-RPC Extension • Software instatiates a LocalInformationService: • infoSvc = LocalInformationService([iniFile]) • infoSvc = LocalInformationService(URL) • Any calls to get(…)which do not find a (section,option) pair in the locally loaded information will refer to a remote InformationService XML-RPC server • Local values override remote (global) values • Extended functionality: • void copy(section) • void copyAll() • list = getsection(section)

  9. Code Example [DEFAULT] MasterInfoServiceURL = http://www.example.com:9130 [ServiceA] ServiceName = “The Big A Service” class ServiceA: def __init__(self): otherServices = [‘sB’,’sC’] self.infoSvc = LocalInformationService(‘serviceA.ini’) masterIS = self.infoSvc.get(‘DEFAULT’,’MasterInfoServiceURL’) name = self.infoSvc.get(‘ServiceA’,’ServiceName’) for s in otherServices: self.infoSvc.setIS(master) svcURL = self.infoSvc.get(s,’ServiceURL’) self.infoSvc.setIS(svcURL) self.infoSvc.copy(s) self.connectToService(s)

  10. XMPP for Service and Agent Monitoring • Extensible Messaging and Presence Protocol (XMPP) grew out of Jabber Instant Messaging framework • Provides: • Asynchronous messaging queues • Broadcast messaging • Presence awareness • Publish/Subscribe framework • RPC mechanisms

  11. XMPP Architecture XMPP-Core IETF Draft XMPP-IM IETF Draft Server: routes and buffers messages, handles presence probes S C S Client: thin client, only responsible for processing incoming messages C C C DB C C DB DB: Stores user profiles and rosters

  12. XMPP Architecture (con’t) • Server takes all responsibility • Client only requires “user” to provide authentication details • All other user details are stored on the server • Makes for very light client • <XML> based and Connection based • Client opens an XML stream with Server • Events are passed as XML fragments • Called Stanzas in XMPP parlance

  13. Message Types • XMPP only supports three types of XML Messages: • <message> : push based general communication • Instant message • One to one chat • Group chat (broadcast) • <presence> : availability and status • <iq> : pull based RPC mechanism

  14. Current Usage • DIRAC Services are loosely coupled by using asynchronous event model • Services connect to Jabber server • A service registers message handlers for the various affiliated services • A Service can communicate with other services by sending messages • DIRAC Resiliance to Service Failure • Jabber servers are mature and stable • Queues messages if service unavailable • Service retreives messages on restart

  15. Future Plans • Monitor Agents and Services via XMPP <presence> rosters • Allow Service access via XML-RPC over XMPP • Communicate directly with jobs on WNs via <iq> • Only outbound IP connectivity is required • Messages are tunneled back over open stream • Great potential for interactive jobs

More Related