1 / 17

The Use of Condor in the gLite Grid Middleware

The Use of Condor in the gLite Grid Middleware. Erwin Laure Condor Week 14-15 March 2005. Contents. Overview on EGEE and gLite gLite and Condor Future plans. The EGEE Project. EU funded (2 years until March 2006)

timothy-brock
Download Presentation

The Use of Condor in the gLite Grid Middleware

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Use of Condor in the gLite Grid Middleware Erwin Laure Condor Week 14-15 March 2005

  2. Contents • Overview on EGEE and gLite • gLite and Condor • Future plans Condor Week 2005

  3. The EGEE Project • EU funded (2 years until March 2006) • EGEE offers the largest production grid facility in the world open to many applications (HEP, BioMedical, generic) • Existing production service based on LCG • Next generation open source web-services middleware being re-engineered taking into account production/ deployment/ management needs • Well-defined, distributed support structure to provide eInfrastructure that is available to many application domains • Middleware Activity • Re-engineer and harden Grid middleware • Provide production quality middleware Collaborations Global Grid Operations, Support and training Network infrastructure(GÉANT) www.eu-egee.org Condor Week 2005

  4. EGEE Activities • 48 % service activities (Grid Operations, Support and Management, Network Resource Provision) • 24 % middleware re-engineering (Quality Assurance, Security, Network Services Development) • 28 % networking (Management, Dissemination and Outreach, User Training and Education, Application Identification and Support, Policy and International Cooperation) Emphasis in EGEE is on operating a production grid and supporting the end-users Condor Week 2005

  5. Computing Resources: Feb 2005 Country providing resources Country anticipating joining In LCG-2: • 113 sites, 30 countries • >10,000 cpu • ~5 PB storage Includes non-EGEE sites: • 9 countries • 18 sites Condor Week 2005

  6. gLite Grid Middlewareguiding principles • Service oriented approach • Allow for multiple interoperable implementations • Lightweight (existing) services • Easily and quickly deployable • Use existing services where possible • Condor, EDG, Globus, LCG, … • Portable • Being built on Scientific Linux and Windows • Security • Sites and Applications • Performance/Scalability & Resilience/Fault Tolerance • Comparable to deployed infrastructure • Co-existence with deployed infrastructure • Co-existence with LCG-2 and OSG (US) are essential for the EGEE Grid services • Site autonomy • Reduce dependence on ‘global, central’ services • Open source license Condor Week 2005

  7. LCG-2 (=EGEE-0) 2004 prototyping prototyping product 2005 product LCG-3 EGEE-1 From Development to Product • Fast prototyping approach • Small scale testbed (initially CERN and Wisconsin) • Single out individual components for deployment on pre-production service (originally LCG-2/EGEE0 based) • These components need to go through integration and testing • To ensure they are deployable and basically work Condor Week 2005

  8. gLite Services and Responsible Clusters JRA3 UK Access Services Grid AccessService API CERN IT/CZ Security Services Authorization Information & Monitoring Services ApplicationMonitoring Information &Monitoring Auditing Authentication Data Services Job Management Services MetadataCatalog File & ReplicaCatalog JobProvenance PackageManager Accounting StorageElement DataManagement WorkloadManagement ComputingElement Site Proxy Condor Week 2005

  9. gLite Services for Release 1 JRA3 UK Access Services Grid AccessService API CERN IT/CZ Security Services Authorization Information & Monitoring Services Application Monitoring Information &Monitoring Auditing Focus on key servicesRelease date is March 31st 2005 Authentication Data Services Job Management Services MetadataCatalog File & ReplicaCatalog JobProvenance PackageManager Accounting StorageElement DataManagement WorkloadManagement ComputingElement Site Proxy Condor Week 2005

  10. Condor and gLite • Design team including representatives from Middleware providers (AliEn, Condor, EDG, Globus,…) including US partners produced middleware architecture and design. • Takes into account input and experiences from applications, operations, and related projects • DJRA1.1 – EGEE Middleware Architecture (June 2004) • https://edms.cern.ch/document/476451/ • DJRA1.2 – EGEE Middleware Design (August 2004) • https://edms.cern.ch/document/487871/ • Wisconsin is one of the sites of the development prototype • Using Condor pool as backend • Using Globus RLS • Use VDT distribution of Condor and Globus Condor Week 2005

  11. WMS Condor-C Condor Week 2005

  12. Gatekeeper LCASLCMAPS WSS LaunchCondor-C LaunchCondor-C The current gLite CE • Collaboration of INFN, Univ. of Chicago, Univ. of Wisconsin-Madison, and the EGEE security activity (JRA3) Submitjob CEMon Notifications Blahpd Condor-C CE Localbatchsystem LSF Condor PBS/Torque Condor Week 2005

  13. Catalog Catalog Catalog MOM MOM MOM LocalCat LocalCat LocalCat Data Management Services • Efficient and reliable data storage, movement, and retrieval on the infrastructure • Storage Element • Reliable file storage (SRM based storage systems) • Posix-like file access (gLite I/O) • Transfer (gridFTP) • File and Replica Catalog • Resolves logical filenames (LFN) to physical location of files (URL understood by SRM) and storage elements • Hierarchical File system like view in LFN space • Single catalog or distributed catalog (under development) deployment possibilities • File Transfer and Placement Service • Reliable file transfer and transactional interactions with catalogs • Stork being evaluated • Data Scheduler • Scheduled data transfer in the same spirit as jobs are being scheduled taking into account e.g. network characteristics (collaboration with JRA4) • Under development • Metadata Catalog • Limited metadata can be attached to the File and Replica Catalog • Interface to application specific catalogs have been defined Data Scheduler VOs Catalog Catalog Catalog Site boundary VOs FPS FPS FPS FPS TransferAgent SRM GridFTP I/O Storage Element Condor Week 2005

  14. Evolutions foreseen in 2005 Here follows a list of main topics we still need to address, details and other topics need to be worked out with operations and applications • WMS • WS Interface • Better support for bulk job submission • CE • Head node monitoring • Guard and if necessary pause/resume services running on the head node • SUDO service • Currently one Condor-C instance needed per user • Condor-C should run under a VO user and submit jobs via a sudo service to the local batch system • This is being done in collaboration with Condor and Globus Condor Week 2005

  15. Evolutions foreseen in 2005 • Catalogs • Distributed and single deployment options • FTS/FPS • Channel management • Clarify the role of Stork • Data Scheduler • “broker” for data transfer “jobs” • R-GMA Information system • Web service version • Package Manager Second major gLite release foreseen at the end of 2005 Condor Week 2005

  16. Summary • Contributions of Condor to gLite span whole process • Design – prototyping – product • International collaborations in Grid middleware are essential • Grid middleware cannot be developed within a single corner • Effective exchange of ideas, requirements, solutions and technologies • Coordinated development of new capabilities • Open communication channels • Early detection of differences and disagreements • Condor link to OSG is very important to gLite • A common integration and testing project is being defined • gLite process, in particular through the strong interactions between European (EGEE) and US (Condor and Globus) projects is a first step towards truly international collaborative middleware development Condor Week 2005

  17. More information http://www.glite.org http://cern.ch/egee-jra1 Condor Week 2005

More Related