R-GMA: Production Services for Information and Monitoring in the Grid John Walk / RAL

1. R-GMA: Production Services for Information and Monitoring in the GridJohn Walk / RAL UK e-Science AHM2004, Nottingham www.eu-egee.org EGEE is a project funded by the European Union under contract IST-2003-508833

2. What’s the talk about? • Overview of R-GMA • Practical experience with R-GMA • Building a production-quality Web Services version of R-GMA

3. Who are we? • The system (R-GMA) • Grid Information and Monitoring System created in the European DataGrid (EDG) project • Now being taken forward as part of the production-quality middleware being developed in the EGEE project • The people (JRA1-UK) • 9 Full-time at Rutherford Appleton Laboratory (CCLRC) • 1 Full-time at Queen Mary, University of London • Contributions from Heriot-Watt University (Edinburgh) and Trinity College Dublin • We can be e-mailed at: • jra1-uk@physics.gla.ac.uk

4. Relational Grid Monitoring Architecture • An implementation of the GGF Grid Monitoring Architecture (GMA). Publish Data Producer application Producer Service API Register Registry Service Data Query Locate Mediator Send Query Consumer application Consumer Service API Receive Data

5. Relational Grid Monitoring Architecture • An implementation of the GGF Grid Monitoring Architecture (GMA) • All data modelled as tables: a single schema gives the impression of one (loosely coupled) virtual database for VO Publish Tuples Producer application Producer Service API Register SQL “INSERT” Registry Service Tuples Query SQL “SELECT” Locate Mediator Send Query Consumer application Consumer Service API Receive Tuples Schema Service SQL “CREATE TABLE”

6. Secondary Producers • Used to republish data • To co-locate information to speed up queries • To reduce network traffic • To use different producer properties PP PP SP PP PP – Primary Producer SP – Secondary Producer PP

7. A unified approach to information and monitoring • Provides information on both: • The Grid itself • Mainly for the middleware packages, but the user may also query it to understand the status of the Grid • Grid applications • For users • Flexible infrastructure • All data has a timestamp enabling its use for monitoring • Dynamic addition and deletion of information producers • Dynamic creation and deletion of tables in schema • Able to cope with failures in a distributed environment

8. Resilience • Soft state registration protects services and registry from defunct producers and consumers • Registry and schema are replicated to avoid single point of failure (and improve scalability) • All services tolerate failure of remote service or resource by seeking an alternative, trying again later, or failing gracefully if these are not possible

9. Grid Application: CMS Job Monitoring http://cmsinfo.cern.ch/

10. BOSS Job wrapper Monte Carlo Simulation Job P BOSS Database BOSS Job wrapper Monte Carlo Simulation Job P P R-GMA Primary Producer C R-GMA Consumer R-GMA in CMS In testing… • Target: 2000-job batch • Exceeded in 2004 C

11. Grid Middleware: Network Monitoring http://ccwp7.in2p3.fr/

12. GridFTP Log P R-GMA Primary Producer SP R-GMA Secondary Producer R-GMA in EDG WP7 Network Monitoring Network metrics include… Network Managers Resource Brokers Replica Managers TCP throughput SP MySQL Database ICMP packet loss Transfer times P P PingER IPerf UDPmon Network Monitoring Nodes Storage Elements

13. Enabling Grids for E-science in Europe “EGEE will integrate current national, regional and thematic Grid efforts to create a seamless European Grid infrastructure for the support of the European Research Area” (http://www.eu-egee.org) • R-GMA is the Information Services component of the gLite middleware being developed for EGEE • Main efforts are: • Producing production-quality software • Migrating to Web Services • Improving portability • Duration of project is two years, with first release of software due after first year (March 2005)

14. Main achievements so far (JRA1-UK) • Specification complete • Design complete • WSDL produced for user interfaces • Prototype of new API available (Java, C, C++, Python), as a wrapper around the old one (feedback welcome) • Registry replication completed; outline design for schema replication produced

15. Next steps • Complete move to WS-I compliant Web Services • Introduce multi-VO support • Complete security design work • Complete the schema replication • Port to Win32 (EGEE secondary platform) • Watch developments elsewhere and seek to inter-work (e.g. contribution to OGSA; collaboration with L&B work)

16. R-GMA Web Services AXIS SERVLET • API available for Java, C, C++ and Python • Users may by-pass API if they wish, but API is the easiest way to use R-GMA services • Registry and Schema also implemented as Web Services, but not accessed directly by users, except for administration SOAP/HTTP Messages Primary Producer Service User Application R-GMA API SOAP STUB Secondary Producer Service On-demand Producer Service SOAP/HTTP Messages And SOAP Faults Consumer Service Created from WSDL using gSOAP or Axis Tomcat Servlet Container

17. R-GMA Security • Many issues are EGEE-wide and are being addressed by JRA3 (Security) • User requirements vary widely from HEP (would probably do without it) to BioMed (very stringent confidentiality requirements) • Will allow installations to turn security off where performance is paramount • Some R-GMA users require access to data to be controlled down to sub-table level. Proposal is to use parameterized views, e.g. SELECT * from Job where Owner=[DN] : DN=[DN];

18. Further Information • JRA1-UK web site • http://hepunx.rl.ac.uk/egee/jra1-uk/ • E-mail • jra1-uk@physics.gla.ac.uk Thanks to the EU and our national funding agencies for their support of this work