1 / 17

The gLite middleware distribution

The gLite middleware distribution. OSG Consortium Meeting Seattle, 21-23 August 2006. Outline. Background and approach adopted Architecture Software process Status Summary. Background & Approach. gLite

monty
Download Presentation

The gLite middleware distribution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The gLite middleware distribution OSG Consortium Meeting Seattle, 21-23 August 2006

  2. Outline • Background and approach adopted • Architecture • Software process • Status • Summary OSG Consortium Meeting - Seattle - 21-23 August 2006

  3. Background & Approach • gLite • Exploit experience and existing components from VDT (Condor, Globus), EDG/LCG, AliEn, and others • Develop a lightweight stack of generic middleware useful to EGEE applications (HEP and Biomedics are pilot applications). • Pluggable components – cater for different implementations • Follow SOA approach, WS-I compliant where possible • Focus is on re-engineering and hardening • Business friendly open source license • Plan to switch to Apache-2 OSG Consortium Meeting - Seattle - 21-23 August 2006

  4. GIN Service Oriented Architecture • gLite follows a Service Oriented Architecture • Facilitate interoperability among Grid services • Allow easier compliance with upcoming standards • The services work together in a concerted way but can also be deployed and used independently, allowing their exploitation in different contexts • Services communicate through the exchange of messages • Slowly moving to WS-* interfaces • Still missing a real standard. Many WS-* specifications • Activity inside GGF-GIN OSG Consortium Meeting - Seattle - 21-23 August 2006

  5. Middleware structure • Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware • Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory • Foundation Grid Middleware will be deployed on the EGEE infrastructure • Must be complete and robust • Should allow interoperation with other major grid infrastructures • Should not assume the use of Higher-Level Grid Services Applications Higher-Level Grid Services Workload Management Replica Management Visualization Workflow Grid Economies ... Foundation Grid Middleware Security model and infrastructure Computing (CE) and Storage Elements (SE) Accounting Information and Monitoring OSG Consortium Meeting - Seattle - 21-23 August 2006

  6. gLite Grid Middleware Services Access CLI API Security Information & Monitoring Authorization Auditing Information &Monitoring Application Monitoring Authentication Data Management Workload Management MetadataCatalog File & ReplicaCatalog JobProvenance PackageManager Accounting StorageElement DataMovement ComputingElement WorkloadManagement Site Proxy Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf OSG Consortium Meeting - Seattle - 21-23 August 2006

  7. Resource usage by VO, group or single user Resource metering: sensors running on resources to determine usage Pricing policies: associate a cost to resource usage if enabled allowed market-based resource brokering privacy: access to accounting data granted only to authorized people (user, provider, VO manager) Basic functionality in APEL, full functionality in DGAS Grid foundation: Accounting OSG Consortium Meeting - Seattle - 21-23 August 2006

  8. WN Grid foundation: Computing Element The CE software • accepts batch jobs (and job control requests) through a gatekeeper • LCG-CE (GT2 GRAM + GSI-enabled Condor) • gLite-CE (GSI-enabled Condor-C) • CREAM (WS-I based interface) • performs the necessary AAA operations and map to a local user • through LCAS/LCMAPS and the GRAM or glexec • passes the job to a layer that interacts with the local resource manager • BLAH • monitors the status of the jobs and reports it to the client • Condor • CEMon (in CREAM) • Web service interface to the CE info Client Grid Computing Element Monitoring Site AAA and local mapping Job Controller LRMS OSG Consortium Meeting - Seattle - 21-23 August 2006

  9. Site File Name (SFN): identifies a Storage Element and the logical name of the file inside it Physical File Name (PFN): argument of file open Storage Resource Manager (SRM) hides the storage system implementation (disk or active tape) checks the access rights to the storage system and the files translates SFNs to PFNs disk-based: DPM, dCache; tape-based: Castor, dCache File I/O: posix-like access from local nodes or the grid GFAL Grid foundation: Storage Element OSG Consortium Meeting - Seattle - 21-23 August 2006

  10. Encrypted Data Storage encrypt and decrypt data on-the-fly Key-store: Hydra N instances: at least M (<N) need to be available for decryption fault tolerance and security Demonstrated with the SRM-DICOM demo at EGEE Pisa conference (Oct’05) High Level Services: EDS Will be DPM (now d-Cache) Will be LFC Will be GFAL OSG Consortium Meeting - Seattle - 21-23 August 2006

  11. Resource brokering, workflow management, I/O data management Web Service interface: WMProxy Task Queue: keep non matched jobs Information SuperMarket: optimized cache of information system Match Maker: assigns jobs to resources according to user requirements Job submission & monitoring Condor-G ICE (to CREAM) External interactions: Information System Data Catalogs Logging&Bookkeeping Policy Management system (G-PBox) High Level Services: Workload Manag. OSG Consortium Meeting - Seattle - 21-23 August 2006

  12. High Level Services: Job Information • Logging and Bookkeeping service • Tracks jobs during their lifetime (in terms of events) • Job Provenance stores long term job information • Supports job rerun OSG Consortium Meeting - Seattle - 21-23 August 2006

  13. gLite Software Process JRA1 Development Directives Error Fixing Software Serious problem SA3 Integration SA3 Testing & Certification SA1 Pre-Production Deployment Packages Testbed Deployment Problem Fail SA1 Production Infrastructure Pre-Production Deployment Fail Integration Tests Pass Functional Tests Pass Fail Installation Guide, Release Notes, etc Scalability Tests Release Pass OSG Consortium Meeting - Seattle - 21-23 August 2006

  14. gLite Software Process • Technical Coordination Group (TCG) • gathers & prioritizes user requirements from HEP, biomed, (industry), sites • gLite development is client-driven! • Software from EGEE-JRA1 and other projects • JRA1 preview testbed (currently being set up) • early exposure to users of “uncertified” components • SA3 Integration Team • Ensures components are deployable and work • Deployment Modules implemented high-level gLite node types • (WMS, CE, R-GMA Server, VOMS Server, FTS, etc) • Build system now spun off into the ETICS project (Jan 2006) • SA3 Certification Team • Merge of the JRA1 testing and SA1 certification teams • Dedicated testbed; test release candidates and patches • Develop test suites • SA1 Pre-Production System • Scale tests by users OSG Consortium Meeting - Seattle - 21-23 August 2006

  15. LCG-2 gLite 2004 prototyping prototyping product 2005 product 2006 gLite 3.0 gLite status • Convergence of LCG 2.7.0 and gLite 1.5.0 in spring 2006 • continuity on the production infrastructure ensured usability by experiments • new features from gLite 1.5.0 • Current activities • Improve usability, efficiency and performance • Migration to VDT 1.3.11 (GT4 pre-WS) • Support for Scientific Linux 4 and 64-bit • Support for other platforms will follow • New data management components for Biomed applications on the production infrastructure • Certify new components (CREAM, Job Provenance, GPBox, ...) • Interoperation with other projects and adherence to standards • Open source (Apache) license OSG Consortium Meeting - Seattle - 21-23 August 2006

  16. Summary • gLite 3 is an important milestone in EGEE program • New components from gLite 1.5 being deployed for the first time on the Production Infrastructure • Address requirements in terms of functionality and scalability • Components deployed for the first time need extensive testing! • New organization in EGEE II • more controlled software process • development is client driven (TCG) • Development is continuing to provide increased robustness, usability and functionality • Collaboration with other projects for interoperability and definition/adoption of international standards OSG Consortium Meeting - Seattle - 21-23 August 2006

  17. www.glite.org OSG Consortium Meeting - Seattle - 21-23 August 2006

More Related