R-GMA: today and tomorrow 28/11/2002

1. R-GMA: today and tomorrow28/11/2002 Steve Fisher / RAL <s.m.fisher@rl.ac.uk>

2. Why? • Monitoring and information systems should not be separated • Information may be the latest available • It may be historical • For monitoring • With time stamps • You may want it streamed to you • System should allow you to publish what you want • System should allow you to find out what you want • System should be secure R-GMA today and tomorrow

3. MDS • Monitoring and information systems should not be separated  • Information may be the latest available  • It may be historical  • For monitoring • With time stamps • You may want it streamed to you  • System should allow you to publish what you want  • System should allow you to find out what you want  • System should be secure  R-GMA today and tomorrow

4. R-GMA – What? • Use the GMA from GGF • A relational implementation • Applied to both information and monitoring • Creates impression that you have one RDBMS per VO Producer store Registry lookup Consumer R-GMA today and tomorrow

5. Relational Approach • Not a general distributed RDBMS system, but a way to use the relational model in a distributed environment where ACID properties are not generally important. • Producers announce: SQL “CREATE TABLE” publish: SQL “INSERT” • Consumers collect: SQL “SELECT” R-GMA today and tomorrow

6. Not just one Producer • DataBaseProducer (J) • Relatively slow • Information not lost • Clean up strategy needed (J+30 – being worked on now) • Streaming not supported (nor defined) • Supports joins • CircularBufferProducer (being deprecated) (J) • Fast • Uses an SQL parser – no RDBMs involved • Information will be lost if not consumed in time • Streaming supported • 1 write pointer for buffer and 1 read pointer for each Consumer R-GMA today and tomorrow

7. Not just one Producer • StreamProducer (J+30 – ready now) • Fast • Uses an SQL parser – no RDBMs involved • Holds data in memory • Does not support joins • Can define minimum retention period • Resilient Producer (J+30 – ready now) • Like the StreamProducer but won’t lose data if system crashes • LatestProducer (J+30) • Just holds the latest information for any “primaryish” key • Supports joins R-GMA today and tomorrow

8. Producer Inheritance • This is not visible to the user Concrete Class (an example – some classes inherit from lower down) DataBaseProducer Cleanable Supports clean up mechanism Insertable Allows rows to be inserted Declarable Allows tables to be declared Methods shared by all our APIs - e.g. disconnect() APIBase R-GMA today and tomorrow

9. CreateTable, Port, Protocol, Security, SQL Support, Multiple Query Support Register CP API Canonical Producer Servlet Security Insert User Code Query Port Files Canonical Producer • Allows user defined back end to be invoked • In collaboration with CrossGrid • J+30 – ready now R-GMA today and tomorrow

10. Archiver (J) • It is a combined Consumer-Producer • It is a republisher • …but we call it an Archiver • You just have to tell it what to collect and it does so on your behalf R-GMA today and tomorrow

11. Archiver of A-H Archiver of A-H StreamProducer BS LatestProducer StreamProducer Archiver of I-N Archiver of I-N StreamProducer BS StreamProducer LatestProducer Archiver of O-Z StreamProducer Archiver of O-Z LatestProducer Smith (Wants to be told of each change of state of his job) Fitzwilliam (Wants to look at current state of all his jobs) R-GMA for middleware: L&B • Each Bookkeeping Server publishes to a StreamProducer • Archiver has a where clause to collect jobs belonging to a subset of users • Most queries will be satisfied by one Archiver Consumer Consumer R-GMA today and tomorrow

12. StreamProducer NM WP7 • Each NetworkMonitor publishes to a CircularBufferProducer • Archiver collects all information needed to satisfy the cost function query which WP7 provides for WP2 • Offers a number of interesting queries to deduce information for paths which are not measured StreamProducer NM WP7 cost fn Consumer Archiver GIN SE GIN CE R-GMA today and tomorrow

13. A user application: CMS • BOSS for job tracking on local farm • It currently forks the executable and parses stdout to publish info directly to an SQL DB • They publish to one table per job type and one table which is common to all job types • This will now publish via R-GMA instead • Providing a scaleable Grid solution R-GMA today and tomorrow

14. R-GMA for end users • GRM/PROVE for parallel applications • Users will be able to publish “Anything” with R-GMA • once security is in place R-GMA today and tomorrow

15. R-GMA from user perspective • APIs in “all” languages • Java, C++, C, Python and Perl • Easy installation and configuration • For developers • Installers • Users • Pulse • GUI to browse (and manipulate) R-GMA data • Browser (via web) • Table d’hote common queries • A la carte R-GMA today and tomorrow

16. R-GMA – How? • Currently based on servlet technology • Tomcat • Multiple hand crafted APIs • Soft state registration • Uniform exception handling • To ensure that useful messages and stack traces are preserved. • Have just started the migration to web and grid services • Apache axis • WSDL generated APIs • Will provide a wrapper for backwards compatibility R-GMA today and tomorrow

17. Application Code Consumer Servlet Consumer API Registry API Registry Servlet Schema API Schema Servlet Producer API Registry API Sensor Code ProducerServlet R-GMA • API – Servlet communication • http(s) in • XML back R-GMA today and tomorrow

18. Schema & Contributions R-GMA today and tomorrow

19. Contributions are Views SELECT * FROM cpuLoad WHERE country = ’UK’ AND site = ’RAL’ SELECT * FROM cpuLoad WHERE country = ’UK’ AND site = ’GLA’ R-GMA today and tomorrow

20. The mediator Sa relational schema (for a virtual database) qqueries posed againstS pProducers, associated with views onS. Currently views have the form: SELECT * FROM r WHERE < ??? > • The Mediator • how to match q with the p’s ? • It is the mediator which makes R-GMA work • It is hidden inside the ConsumerServlet R-GMA today and tomorrow

21. Mediator for the “J” release • Producers register which partition they have by means of a partitioning predicate • A view on a virtual table • The mediator is hidden inside the Consumer • but is an essential part of R-GMA • The final mediator will take “any” SQL statement and from the information in the registry find the right producers • We can now merge information from several producers R-GMA today and tomorrow

22. OGSIfication • We have started to develop demos of web services and grid services • We may have a grid service based schema for J+30 – but it will be wrapped so the user won’t know. R-GMA today and tomorrow

23. Application Code Consumer Servlet Consumer API Registry API Registry Servlet Schema API Schema Servlet Producer API Registry API Sensor Code ProducerServlet R-GMA - OGSIfication • API – Servlet communication • http(s) in • XML back R-GMA today and tomorrow

24. Step 1 - Isolate Servlets Application Code • API – Servlet communication • http(s) in • XML back Consumer Instance Consumer API Registry API Registry Schema API Producer API Registry API Schema Producer Instance Sensor Code R-GMA today and tomorrow

25. Step 2 - Web Services Consumer “Factory” Application • API – derived from WSDL • Web Services • WSDL, SOAP • Issues • context to access instance • HTTP Streaming Consumer Instance Consumer API PortTypes PortTypes Registry PortTypes PortTypes PortTypes Producer API PortTypes Producer Instance Schema Sensor Producer “Factory” R-GMA today and tomorrow

26. Registry Step 3 - OGSA Consumer Factory Application • All Grid Services • OGSA Factories, GSH, GSR • Registry includes HandleMapper • SQL as Service Data Element Query Language • lightweight api causes issues with lifetime management • TerminationInterval then instance loopback to setTerminationTime. Consumer API Consumer Instance Producer API Producer Instance Schema Sensor Producer Factory R-GMA today and tomorrow

27. Other OGSIfication issues • Consider XML as internal representation of service data elements • Depends on other developments • Consider Xquery as service data elements query language • Depends on how Xquery develops • Security • Authorisation looks hard • Registry discovery • same issues as OGSA in general R-GMA today and tomorrow

28. Security • Should be in place for J+30 • Using edg-security for authorisation • Gives secure socket factory for https • Plan to use VOMS R-GMA today and tomorrow

29. Registry & schema distribution • Will have one logical registry and schema per VO (J+30) • Each logical registry will have multiple physical “copies” (J+30) Prod1 ProdServlet1 Registry1 Info mastered by Registry1 Copy of info from Registry2 Prod2 ProdServlet2 Registry2 Info mastered by Registry2 Copy of info from Registry1 R-GMA today and tomorrow

30. Displays • Pulse • Java client application • BrowserServlet • JSP application R-GMA today and tomorrow

31. When? • In the 23rd month of the project R-GMA still not deployed! • We have been ready for months but we are waiting for “J” • We need exposure and field testing • So will be deploying widely in the UK • This will go out at J+3 to the UK • We are only adding and not changing anything so it should have no negative impact R-GMA today and tomorrow

32. GIIS GRIS GRIS GRIS InfoProvider InfoProvider InfoProvider Release 1.2 RB R-GMA today and tomorrow

33. RB GOUT Archivers and other R-GMA components GIN GIN GIN InfoProvider InfoProvider InfoProvider Release J (on dev testbed) Multi-valued attributes make it not totally trivial Archiver’s DB in GOUT cleaned by ad-hoc code R-GMA today and tomorrow

34. GOUT Archiver Consumer (CE) Consumer (SE) Consumer LDAP DataBaseProducer RDBMS Consumer (..) Clean up This will be much simpler with the J+30 release R-GMA today and tomorrow