1 / 17

The gLite middleware Overview

Pete Gronbech Oxford University. The gLite middleware Overview. The purpose of this middleware section is to describe the key components of the EGEE/gLite model. What the various components do and how they connect: WN, CE, SE, WMS, LFC, FTS …. gLite 3.1. The current release is gLite 3.1

quasim
Download Presentation

The gLite middleware Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pete Gronbech Oxford University The gLite middleware Overview

  2. The purpose of this middleware section is to describe the key components of the EGEE/gLite model. What the various components do and how they connect: WN, CE, SE, WMS, LFC, FTS …

  3. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 gLite 3.1 The current release is gLite 3.1 It is updated almost every week Its purpose is to provide a stable platform for production grid usage Some highlights Data Management Workload Management Information System Security and Authorisation

  4. Workload Management • Just getting a job started involves a lot of services VOMS BDII WMS UI MyProxy WN TORQUE CE The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008

  5. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Users • Security • The Grid Security Infrastructure (GSI) is based on X509 certificates issued by Certification Authorities (CA) • Users are grouped in to Virtual Organisations (VOs). The VO Management Service (VOMS) is used to manage information about the roles and privileges of users within a VO. • a Grid proxy is generated by contacting the VOMS server and by default lasts for 12 hours. • If a job does not finish before the expiration time of the proxy it is aborted. Long lived proxies are a security risk. To overcome this, a proxy credential repository system, can store a long-term proxy in a dedicated server (MyProxy server). The WMS can use this to renew the proxy for a submitted job before it expires.

  6. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 User Interface Node (UI) • The access point to the WLCG/EGEE Grid • Any machine where the user has an account and has their certificate installed. • This is where the user can access all the functionalities of the Information, Workload and Data Management systems.

  7. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Job Submission

  8. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 WMS • WMS • The Workload Management System (WMS) is a service responsible for the distribution and management of computational jobs across Grid resources, in such a way that applications are conveniently, efficiently and effectively executed. A job is expressed in a flexible Job Description Language, based on Condor ClassAds. Different types of jobs are supported: simple batch jobs, DAGs, collections, parametric. On top of them other types of jobs, such as MPI and interactive, can be implemented. In order to address scalability and availability, many independent instances of the service can be deployed, also behind a single DNS alias. • A WMS instance interacts with several other services. Tracking job lifetime relies on the Logging and Bookkeeping Service. Information on service availability, resource status and data localization is gathered from appropriate sources, such as Service Discovery, LFC, BDII, RGMA and CEMon. Access to resources is granted by Computing Elements, available in multiple flavours, including the LCG-CE, the ARC-CE and the CREAM-CE. Security related aspects are addressed interacting with VOMS, Proxy Renewal and, in the near future, the new authorization service. • The main interface is compliant with Web Service specifications, easing the integration with other Grid services.

  9. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Computing Element (CE) • This node provides access to the computing resources at a site (ie a cluster ). • A CE includes a Grid Gate (GG) which acts as an interface to the cluster or Local Resource Management System (LRMS). Common batch systems supported are: PBS/Torque, Sun Grid Engine, CONDOR, LSF. • The jobs then run on Worker Nodes (WN) in the cluster. • The current GG in glite3.1 is the LCG-CE. (The glite-ce was dropped). The new CE is called CREAM (Computing Resource Execution And Management).

  10. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Submission through the WMS WMS CREAM CREAM CREAM WMS/ICE • The infrastructure currently runs with the lcg-CE CREAM is the replacement • Scalability • Parameter passing • Direct submission WS interface • job management (inc proxy renewal) • Improved service administration • A BES/JSDL interface is planned • A first release is ready now for production • Job management tasks are delegated to the gLite Workload Management Service • Can now sustain 30k jobs per day under normal operations Direct Job Submission

  11. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Storage Element (SE) • An SE provides uniform access to data storage, from simple disk arrays to tape based MSS. • SEs support different protocols and interfaces. GSIFTP is used for whole file transfers, while local and remote file access is performed by RFIO or gsidcap. • Most storage resources are managed by Storage Resource Manager (SRM), which provides; file migration from disk to tape, file pinning, space reservation etc. • Implementations: Disk Pool Manager (DPM) for disk only, CASTOR for MSS or dCache for both. StoRM can be used to front other File sytems such as GPFS or Lustre.

  12. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 LFC • LFC • The LCG File catalogue (LFC) offers a hierarchical view of files to users, with a UNIX-like client interface. The LFC catalogue provides: • Logical File Name (LFN) to Storage URL (SURL) mappings (via a GUID); • Authorization on its logical namespace. • The LFC relies on a client-server model, using a proprietary socket interface. The LFC server communicates with a database (either Oracle or MySQL), where all the data is stored. The LFC server is accessible on the client side via: • A command line interface; • the C API; • a Python interface and a Perl interface. • Read-only distributed catalog can be achieved by using Oracle streams. Both lcg_utils and GFAL support the LFC as a possible file catalogue.

  13. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 FTS • The gLite File Transfer Service (FTS) is a low-level data movement service for transferring files between Storage Elements. • In addition, it provides features for administration and monitoring of these transfers. • The FTS exposes an interface to submit asynchronous bulk requests and performs the transfers using either third-party GridFTP or SRM Copy. • The FTS servers are typically deployed at (large) sites where there are large amounts of data to be transferred.

  14. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Software Group Manager accounts • VO users get mapped to pool accounts on the actual WN’s. • A special account (or pool of accounts) are used to install software for the VO in a shared software area. This is then advertised by software TAGS by the information service. • Jobs can have requirements in their JDL to only run on sites providing this software. • Currently (I believe) NGS software is manually installed at sites by the local sys admins. • It would really help GridPP affiliate sites if NGS could adopt the concept of SGM accounts.

  15. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 Multiplatform Support • gLite currently runs on SL3 and SL4 ‘generations’, including selected services on 64bit • A release of the WN on SL5 is nearly ready • More to come • Work continues on Debian 4 • Now profiting from earlier work to make the codebase more portable • Our build programme is being overhauled to be more multiplatform friendly

  16. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 gLite Consortium • EGI as currently proposed does not directly cover middleware development • The EGEE PMB recommended the construction of a new organisation capable of guaranteeing the support and evolution of gLite in the EGI timeframe • Would allow gLite to be properly represented within EGI or a UMD project • A proposal for the gLite consortium is currently under discussion

  17. The gLite middleware - GridPP - NGS Meeting Oxford Oct 2008 References • Oliver Keeble EGEE08 talk http://indico.cern.ch/contributionDisplay.py?contribId=88&sessionId=9&confId=32220 • Glite user guide https://edms.cern.ch/file/722398//gLite-3-UserGuide.html • EGEE-III Functional Description of Grid Components and Associated Work Plan Milestone MJRA1.3.1

More Related