1 / 31

The gLite middleware distribution

Oliver Keeble CERN. The gLite middleware distribution. Overview. Why is gLite? What is gLite? Services and status Roadmap Release process The future. Middleware. What’s so great about being in the middle anyway?. Why is gLite?.

hector
Download Presentation

The gLite middleware distribution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Oliver Keeble CERN The gLite middleware distribution

  2. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Overview Why is gLite? What is gLite? Services and status Roadmap Release process The future

  3. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Middleware What’s so great about being in the middle anyway?

  4. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Why is gLite? So we need software to manage the infrastructure, but why does EGEE produce a distribution? Simplify availability of services and updates Provide integrated 'services‘ from components In the absence of standards, promote interoperability Identify endorsed projects at certified versions Provide an entry point for the uninitiated

  5. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Where does gLite begin... and end? gLite is not just somewhere we stick stuff which might be useful for grids Adding something to gLite has a cost for the project Core services Relationship with Applications RESPECT Operational Tools

  6. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN How is gLite put together? gLite is… A distribution With its own eponymous development project

  7. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN gLite 3.1 The current release is gLite 3.1 It is updated almost every week Its purpose is to provide a stable platform for production grid usage Some highlights Data Management Workload Management Information System Security and Authorisation

  8. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN What’s in there?

  9. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN What’s in there… continued

  10. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Running jobs

  11. Workload Management • Just getting a job started involves a lot of services VOMS BDII WMS UI MyProxy WN TORQUE CE The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

  12. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Data Management

  13. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN The future The EGEE/gLite development activity is JRA1 Most of their effort is earmarked for maintenance Logging, monitoring, docs Address the bugs as found on the EGEE Production and Pre-Production Systems Provide support Address the short/medium term requests of the applications as decided by the TMB Improve the manageability of the middleware in terms of deployability, reliability and usability. Provide the internal unit testing of new or modified software Participate in the definition of the gLite releases together with the SA3 activity Maintain a web page with the relevant information needed by middleware users

  14. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Authorisation service • The single biggest chunk of development anticipated for EGEE-III • What will this bring? • Consistency across services • Distributed Policy Management • Auditing and Logging • Most components will have to be adapted

  15. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Authorisation Service

  16. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN CREAM The infrastructure currently runs with the lcg-CE CREAM is the scheduled replacement Computing Resource Execution And Management service Scalability Parameter passing Direct submission WS interface job management (inc proxy renewal) Improved service administration A BES/JSDL interface is planned A first release is ready now for production

  17. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Submission through the WMS WMS CREAM CREAM CREAM WMS/ICE • Job management tasks are delegated to the gLite Workload Management Service • Can now sustain 30k jobs per day under normal operations Direct Job Submission

  18. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Support for pilot jobs A pilot job is a shell which pulls in the ‘real’ job when its execution starts This is now a common way for VOs to manage their workloads As it runs something on behalf of another user, the mapping to local accounts has to be redone; glexec This mapping must be consistent across a site to prevent two different users sharing an account; SCAS

  19. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN gLExec-on-WN LCMAPS SCAS Site Central AuthZ Service LCAS & LCMAPS GUMS & SAZ glexec and SCAS Query for Authorisation decisions on behalf of users Site SAML2-XACML2 Query Protocol

  20. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Encrypted Data Storage • gLite provides solutions for managing sensitive data • One application is medical data management, where DICOM is a standard for image acquisition and analysis • gLite will release new services for this

  21. LSF PBS/Torque Sun Grid Engine Load Leveler Condor The great thing about standards… Nordugrid CREAM ARC EGEE GRAM v2 Unicore OSG DEISA GRAM v4 NAREGI Teragrid Naregi There are as many Computing Interfaces as Batch Systems! The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

  22. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Some standards relevant to gLite Glue2 Encapsulates the ‘accumulated wisdom’ of a number of years of grid operations Fixes problems Better model e.g. relationships between main entities in compute model have been improved Buy-in BES/JSDL Proposed as solutions by OGF Viability has not been established, investigations are ongoing SRM2 The Storage Resource Manager (SRM) is a middleware component whose function is to provide dynamic space allocation and file management on shared storage components on the Grid. The DPM supports SRM2.2 IGTF / x509

  23. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Glue 2

  24. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN How gLite is put together We allow independent things to remain independent We make a gLite release almost every week!

  25. JRA1 SA3 SA1 Patch States In Configuration Configured [yes] Acceptance Criteria Config Required Ready for Integration With Provider Ready For Certification [no] [yes] Open [no] [yes] [yes] Pass Pass Pass In Pre Production In PPS-PDT Certified In Certification [yes] [no] [no] [no] Other Update Available [no] [yes] In Production Rejected Obsolete [ ] Closed States The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

  26. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Testing infrastructure • Framework • SAM, Nagios, Manual Testing • Regression Tests • Extensive use of virtualisation • ETICS • Move testing as far upstream as possible • Deployment tests • Regression tests • Unit tests

  27. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN A gLite 3.1 update announcement This indicates to an administrator which services they should update

  28. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Multiplatform Support • gLite currently runs on SL3 and SL4 ‘generations’, including selected services on 64bit • A release of the WN on SL5 is nearly ready • More to come • Work continues on Debian 4 • Now profiting from earlier work to make the codebase more portable • Our build programme is being overhauled to be more multiplatform friendly

  29. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Challenges for gLite Release mechanisms What is the best way to deliver the software? IPv6 Response to virtualisation Now the resource can adapt to the application, not the other way around Distribution of VMs for services or applications Support for running jobs in VMs Retirement and end-of-life for services and platforms Fight the Complexity Partially a result of heterogeneity Improve Agility

  30. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN gLite Consortium • EGI as currently proposed does not directly cover middleware development • The EGEE PMB recommended the construction of a new organisation capable of guaranteeing the support and evolution of gLite in the EGI timeframe • Would allow gLite to be properly represented within EGI or a UMD project • A proposal for the gLite consortium is currently under discussion

  31. The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN Summary gLite is critical in the realisation of one of the principal goals of EGEE; Maintain, enhance and simplify the use of the production quality computing infrastructure for an increasing range of researchers in diverse scientific fields gLite is enabling grids for e-science in Europe, across a multitude of scientific disciplines While there are plenty of new ideas in the pipeline, the overarching goal is that stated above, and the primary focus is now on consolidation

More Related