R-GMA: First results after deployment

1. R-GMA: First results after deployment Steve Fisher (EDG - WP3) s.m.fisher@rl.ac.uk https://edms.cern.ch/document/376535/

2. Who we are • Heriot-Watt, Edinburgh • Andrew Cooke, Werner Nutt • IBM-UK • James Magowan, (Manfred Oevers), Paul Taylor • INFN • Roberto Barbera, Giuseppe Save, Gennaro Tortone • Queen Mary, University of London • Roney Cordenonsi, (Ari Datta) • CCLRC • Linda Cornwall, Abdeslem Djaoui, Steve Fisher, Robin Middleton • PPARC • Rob Byrom, Laurence Field, Steve Hicks, Manish Soni, Antony Wilson, (Xiaomei Zhu), Jason Leake • SZTAKI, Hungary • Peter Kacsuk, Norbert Podhorszki • Trinity College Dublin • Brian Coghlan, Stuart Kenny, David O’Callaghan, (John Ryan)

3. R-GMA Producer • Uses the Grid Monitoring Architecture from Global Grid Forum • R-GMA is a relational implementation • Applied to both information and monitoring • Creates impression that you have one RDBMS per Virtual Organisation Registry Information flow Meta-data flow Consumer

4. Relational Approach • Not a general distributed RDBMS system, but a way to use the relational model in a distributed environment where global consistency is not important. • Producers announce: SQL “CREATE TABLE” publish: SQL “INSERT” • Consumers collect: SQL “SELECT” • Some producers, the Registry and Schema make use of RDBMS as appropriate – but what is central is the relational model.

5. Producers • DataBaseProducer – Supports History Queries • Information not lost • Supports joins • Clean up strategy • StreamProducer – Supports Continuous Queries • In memory data structure • Can define minimum retention period • ResilientStreamProducer – Supports Continuous Queries • Like the StreamProducer but won’t lose data if system crashes • So slightly slower • LatestProducer – Supports Latest Queries • Just holds the latest information for any “primaryish” key • Supports joins • CanonicalProducer – Supports anything • Offers anything as relations

6. Archiver (Re-publisher) • It is a combined Consumer-Producer • You just have to tell it what to collect and it does so on your behalf • Re-publishes to any kind of “Insertable” (i.e. not to the CanonicalProducer)

7. Schema & Contributions

8. The Mediator • Producers, associated with views on a virtual data base. • Queries posed against the virtual data base • The Mediator must: • find the right Producers • combine information from them • Can now merge information from several producers • The final mediator will take “any” SQL statement and do the right thing

9. R-GMA Tools • R-GMA CLI • Command Line Interface (similar to MySQL) • Supports single query and interactive modes • R-GMA Browser • JSP application dynamically generating web pages • Supports pre-defined and user-defined queries • Pulse • R-GMA Java client-based GUI • Supports streaming and simple graphical displays

10. A user application: CMS • BOSS for job tracking on local farm • It currently forks the executable and parses stdout to publish info directly to an SQL DB • They publish to one table per job type and one table which is common to all job types • They are now ready to publish via R-GMA instead • Providing a scaleable Grid solution

11. GIN and GOUT (Gadget IN and Gadget OUT) Archiver LDAP InfoProvider Consumer (CE) ConsumerAPI DataBase Producer Consumer (SE) GIN RDBMS Consumer (SiteInfo) R-GMA CircularBuffer Producer GOUT CircularBuffer Producer LDAP Server R-GMA Consumers GIN LDAP InfoProvider

12. CE and SE Tables “Select a ComputingElement with at least 1 free CPU that also has a CloseStorageElement with at least 1000 MB of free space” SELECT DISTINCT ComputingElement.CEId FROM ComputingElement, CloseStorageElement,StorageElementStatus WHERE ComputingElement.FreeCPUs > 0 AND (ComputingElement.CEId = CloseStorageElement.CEId AND CloseStorageElement.CloseSE = StorageElementStatus.SEId AND StorageElementStatus.SEfreespace > 1000)

13. Registry OGSIfied R-GMA Consumer Factory Application • All Grid Services • OGSA Factories, GSH, GSR • Registry includes HandleMapper • SQL as Service Data Element Query Language Consumer API Consumer Instance Producer API Producer Instance Schema Sensor Producer Factory

14. Other technicalities – no time today • Soft-state Registration and the Registry • Registry records existence of Producers and Consumers • Registry holds last contact time and ‘expiry’ time • Producers and Consumers periodically refresh their time stamps • Scheduled removal of entries that have timed-out • Registry & schema distribution • Will have one logical registry and schema per VO • Each logical registry will have multiple physical “copies” • Self healing algorithm • Security • etc …

15. Performance • By design: • Very flexible - to avoid bottlenecks • Powerful queries allow a single query to be made • Performance and Optimisation • Use NetLogger and profiling tools to identify possible bottlenecks

16. Results • It has only just been deployed in the EDG development testbed and we do not yet have the results which the title of this talk implied.

17. Summary and the future • R-GMA is a combined Grid information and monitoring system • Just deployed in the EDG development testbed • Focusing on reliability, stability and performance for the rest of the project (9 months) Thanks to the EU and our national funding agencies for their support of this work