1 / 26

SG-DG Bridges Zoltán Farkas, MTA SZTAKI

SG-DG Bridges Zoltán Farkas, MTA SZTAKI. Outline. Introduction, aims SG features: EGEE DG features: BOINC (and XtremWeb) BOINC -> EGEE bridge 3G Bridge architecture EGEE -> BOINC bridge. Introduction, aims.

skyler-callahan
Download Presentation

SG-DG Bridges Zoltán Farkas, MTA SZTAKI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SG-DG BridgesZoltán Farkas, MTA SZTAKI The EDGeS project receives Community research funding

  2. Outline SG-DG BridgesZoltán Farkas • Introduction, aims • SG features: EGEE • DG features: BOINC (and XtremWeb) • BOINC -> EGEE bridge • 3G Bridge architecture • EGEE -> BOINC bridge

  3. Introduction, aims SG-DG BridgesZoltán Farkas • The EDGeS project aims to offer an infrastructure that integrates Service Grid (SG) and Desktop Grid (DG) infrastructures • Users of one grid type should be able to make use of the other grid type in a transparent way and vice versa • Thus, the integrated infrastructure will offer the advantages of the two grid type • The core component of this infrastructure is the SG-DG bridge technology

  4. SG features – EGEE I. SG-DG BridgesZoltán Farkas • A big set of services: • WMS: broker, scheduling jobs to resources • LB: logging and bookkepping service • BDII: information system • WN: worker node, does actual job execution • CE: computing element, collects WNs in a queue using an LRMS • SE: storage element, used to store large files • LFC: file catalogue, files stored on SE can be organized into a directory structure • MyProxy: proxy certificate storage • VOMS: virtual organization membership handling component • R-GMA & APEL: accounting services

  5. SG features – EGEE II. SG-DG BridgesZoltán Farkas • Mostly institutes provide the computing resources • Resources are organized into Virtual Organizations • Users with a registered certificate accepted in some VO can use the infrastructure • Basically any kind of job can be executed, with some restrictions

  6. SG features – EGEE III. SG-DG BridgesZoltán Farkas

  7. Grid features – BOINC I. SG-DG BridgesZoltán Farkas • One central service per project with limited access that stores work to be processed • Desktop PCs connect with a simple client application and offer their free CPU cycles • Client application fetches workunits, processes them, and uploads results to the server • Mostly the same application is run with many input data sets (parameter study applications)‏

  8. Grid features – BOINC II. SG-DG BridgesZoltán Farkas BOINC Client pool BOINC project server WU/APP/ Result Database Scheduler Work gen. Assim. BOINC Project Admin ...

  9. BOINC -> EGEE I. SG-DG BridgesZoltán Farkas • Task to be solved: • Process BOINC workunits • In the EGEE infrastructure • Develop a bridge that: • Can handle BOINC workunits • And is able to create EGEE jobs from the workunits, and run them in EGEE

  10. BOINC -> EGEEPossible solutions SG-DG BridgesZoltán Farkas • Agent-based execution: • Send BOINC clients to EGEE • BOINC client connects to BOINC server to fetch work and report results • Wrapping workunits execution: • Send BOINC applications to EGEE • Fetch BOINC workunits, and execute them in an EGEE job, finally report results

  11. BOINC -> EGEEFirst version SG-DG BridgesZoltán Farkas fork()‏ Watch fork()‏ Watch EGEE UI Machine EGEE CE Boinc Client JobWrapper Process1 gLite WMS WN WN • Create JDL file, • Submit, • Get status, • Get output. WU slot1 WN WN Jobi+1 JobWrapper ProcessN EGEE CE • Create JDL file, • Submit, • Get status, • Get output. Jobi+N WN WN WU slotN WN WN

  12. Lessons learnt SG-DG BridgesZoltán Farkas • EGEE WMS doesn't like periodic interaction (needs restart every x hours)‏ • Workunits should be gathered • EGEE Operations sometimes fail • If there is a failure, retry the operation at most three times • Ways to improve: • Interact with WMS the least possible times • Handle pack of jobs instead of individual jobs

  13. Improved BOINC → EGEE bridge SG-DG BridgesZoltán Farkas • Collect jobs originating from BOINC: • Place them in a queue • New jobs in the queue are periodically handled by an EGEE plugin, that • Uses Collection possibilities of EGEE to submit many jobs in one request • This way the usage of the WMS is reduced

  14. Improved bridge architecture SG-DG BridgesZoltán Farkas JobWrapper Process1 WU DB EGEE Plugin gLite WMS • Add • Check • Get output WUi+1 Jobi+1 Jobi+1 WUi+2 Jobi+2 WUi+3 JobWrapper ProcessN Jobi+k • Add • Check • Get output WUi+N Jobi+k

  15. Bridge generalisation SG-DG BridgesZoltán Farkas • Jobwrapper → Source grid producers • Produce jobs originating from source grids • WU DB → Job database + Queue Manager • Stores job produced by source grid producers • Selects jobs for execution • EGEE plugin → Destination grid consumers/plugins • Execute jobs in the job database in the supported destination grids

  16. Generic Grid-Grid Bridge (3G Bridge)‏ SG-DG BridgesZoltán Farkas Src Grid1 Src Grid Prod1 Job Database + Queue Manager Dst Grid Cons1 Dst Grid1 Src Grid2 Src Grid Prod2 Dst Grid Cons2 Dst Grid2 Src Gridn Src Grid Prodn Dst Grid Consm Dst Gridm

  17. Job Database + Queue Manager SG-DG BridgesZoltán Farkas Received job storage Job Database Generic interface above grid plugins DC-API Plugin Control path Queue Manager EGEE Plugin GridHandler Interface Job Handler Interface Interface for sources Grid plugin (submit jobs, update status, get output, ...)‏ Scheduler Received job handler, grid plugin user XWeb Plugin

  18. EGEE -> BOINC I. SG-DG BridgesZoltán Farkas • Transparent method for running EGEE jobs on BOINC DGs • User interacts with EGEE using EGEE tools • 3G Bridge used to transfer jobs to BOINC • Special CE created to catch EGEE jobs • EDGeS AR is used to check validity of applications

  19. EGEE job submission SG-DG BridgesZoltán Farkas

  20. EGEE -> BOINC extension key concept SG-DG BridgesZoltán Farkas • Create a new GRAM jobmanager/LRMS: • For every job, we get the job info (executable name, input files used) from the wrapper script submitted by the EGEE WMS • Add the job to the 3G Bridge • Report logging using DGAS/glite-lb-logevent • The 3G Bridge uses a DC-API plugin to run the job on BOINC

  21. 3G Bridge: EGEE → BOINC SG-DG BridgesZoltán Farkas Job Database + Queue Manager EGEE EGEE producer DC-API BOINC

  22. EGEE producerOverview SG-DG BridgesZoltán Farkas • A new GRAM jobmanager • Gets job information from the WMS wrapper script • Checks if exe is a validated one • Checks if exe is supported by one of the attached BOINC (or XtremWeb) projects • Gets files from WMS • Adds job to 3G Bridge job DB • Polls status of jobs in 3G Bridge DB • Gets results from 3G Bridge and uploads to WMS

  23. DC-API plugin SG-DG BridgesZoltán Farkas • Use DC-API to generate BOINC WUs • Jobs are read from the 3G bridge DB • 3G DB entries are updated on events • The plugin has already been implemented for the CancerGrid system

  24. EGEE -> BOINC: Overview of the system SG-DG BridgesZoltán Farkas EGEE BDII BOINC BOINC CE Info provider Report resources and performance DB 3G Bridge DC-API plugin EGEE WMS Submit job LRMS Add job Watch job Logevents Log events Send output EGEE LB EGEE VOMS Check EXE Submit job BOINC client pool Watch Get output X509 proxy EGEE UI EDGeS Application Repository Get EXE

  25. 3G Bridge Data Handling Issues SG-DG BridgesZoltán Farkas • EGEE applications might use huge input files • For data distribution, ADICS/ATTIC can be used (developed by Cardiff University) • 3G Bridge uses ATTIC to publish selected files (recent development) • ATTIC support in DGs (BOINC/XtermWeb) is work in progress

  26. Conclusions SG-DG BridgesZoltán Farkas • The 3G Bridge architecture: • Offers transparent way for running jobs on BOINC for EGEE users • Offers transparent way for running BOINC jobs on the EGEE infrastructure • Has been extended to support P-GRADE Portal parameter study applications (thus special case of remote file handling is solved) • Initial support for handling large amount of data • Existing 3G Bridge plugins: EGEE, DC-API, XtremWeb • Future 3G Bridge plugins: OurGrid

More Related