1 / 18

Bridging EGEE to BOINC and XtremWeb

Bridging EGEE to BOINC and XtremWeb. GIN : From interoperation to interoperability. Bridging EGEE to BOINC and XtremWeb GIN : From interoperation to interoperability Overview. Definitions : Service Grids and Desktop Grids Presentation of the EDGeS project Bridge BOINC  EGEE

nishi
Download Presentation

Bridging EGEE to BOINC and XtremWeb

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bridging EGEEto BOINC and XtremWeb GIN : From interoperation to interoperability The EDGeS project receives Community research funding

  2. Bridging EGEE to BOINC and XtremWebGIN : From interoperation to interoperabilityOverview • Definitions : Service Grids and Desktop Grids • Presentation of the EDGeS project • Bridge BOINC  EGEE • Bridge XtremWeb  EGEE • Bridge EGEE  BOINC • Bridge EGEE  XtremWeb • Architecture of the EDGeS 3G Bridge • Desktop Grid Production Infrastructure • OGF standards used for future interoperability Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  3. Bridging EGEE to BOINC and XtremWeb SG = Service Grid = Managed grid of managed computing clusters VOMS Admin Grid Admin Site Admin VOMS Server Manages VO X509 proxy with VOMS extensions AccountingLogging & Bookkeeping X509 proxy Gives Accounting and Auditing Manages Site Gives Job Status Log Log Publishes available Resources Submits Job with X509 proxy Meta-scheduler(WMS) Site Computing Resource Pushes Job Sends back Output Sandbox Sends back Output Sandbox Accesses Data with X509 proxy Grid User Accesses Data with X509 proxy Site Storage Resource Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  4. Bridging EGEE to BOINC and XtremWeb SG = Service Grid = Managed grid of managed computing clusters • Computing and Storage Resources are managed by trained staff inside Sites and are authenticated by X509 certificates. • Users are authenticated by X509 certificates or proxies. • Users belong to VOs and get a X509 proxy from a VOMS server to :– Access data,– Submit jobs. • Executables are NOT authenticated. So trust is primarily between Sites and VOs. • Order of magnitude is typically 100 000 CPUs. • A meta-scheduler (WMS) pushes the jobs to resources with are both suitable and available. Examples : EGEE, NorduGrid, OSG, DEISA, … Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  5. Bridging EGEE to BOINC and XtremWeb DG = Desktop Grid = Loose grid scavenging idle resources Unit of Work = Application + Input Data Requests Unit of Work Grid Server withApplication Repository Computing Resource (often Desktop Computer) Submits input data for an application Sends Unit of Work Sends back results Sends back results Grid User Owns Resource Certifies Application Accepts or Refusesan applicationon his resource Currently, for BOINC, both roles of ‘Application Manager’ and ‘Grid User’ are fulfilled by ‘BOINC Project Owners’. Resource Owner(often volunteer) Application Manager Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  6. Bridging EGEE to BOINC and XtremWeb DG = Desktop Grid = Loose grid scavenging idle resources • Computing and Storage Resources are owned by various Owners (it is often volunteer computing), but they are NOT managed and NOT authenticated. • Grid Servers are authenticated by a X509 certificate. • Users are authenticated by the Grid Servers, but NOT by the Computing and Storage Resources. • Executables are certified by managers of the Grid Servers. So : – Resource Owners have to trust the Grid Servers, – BOINC sends each Work Unit to several Resource Owners, because BOINC does NOT fully trust them. • Order of magnitude can be 1 000 000 CPUs. • Starving Computing Resources pull Work Units from Grid Servers. Examples : BOINC, XtremWeb, xGrid Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  7. Bridging EGEE to BOINC and XtremWebPresentation of the EDGeS project New FP7 project started on 01/01/2008 • Integrate Service Grids and Desktop Grids • Enable very large number of computing resources (100K-1M processors) • Attract new scientific communities • Provide a Grid application development environment • Provide application repository and bridges for the execution in the SG-DG system WLCG (CERN) gLite (EGEE) ARC (NorduGrid) VDT (OSG) Unicore (DEISA) EDGeS Future Current Boinc (Berkeley) XtremWeb (INRIA/IN2P3) Xgrid (Apple) Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  8. Bridging EGEE to BOINC and XtremWebPresentation of the EDGeS project http://www.edges-grid.eu Now, Interoperation : • Ad-hoc bridges and interfaces between EGEE, BOINC and XtremWeb. • A MoU between EDGeS and EGEE has been signed on 23 Sept 2008. • XtremWeb users must have a X509 certificate, be registered in a VO and submit their Jobs with a X509 proxy. • BOINC Project Owners must have a X509 certificate, be registered in a VO and store a medium-term X509 proxy in a MyProxy server. • All files must be transferred through the Input and Output sandboxes. In the future : • Interoperability using OGF standards, in order to bridge more Grids. • Better support of grid file access (GFAL, lcg_utils and GridFTP). Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  9. Bridging EGEE to BOINC and XtremWebBridge BOINC  EGEE (WU = Work Unit) EDGeS 3G bridge EGEE BOINC Handler1 for each (BOINC server, BOINC Project Owner, EGEE VO) triple Queue Manager & Job DB EGEE Plugin1 for each (BOINC Project Owner, EGEE VO) pair WMS Work Unit BOINC jobwrapper client (simulating a large BOINC computing resource) WUi+1 Job Handler Interface Jobi+1 BOINC Server Grid Handler Interface Jobi+1 WUi+2 3G job-wrapper 3G job-wrapper WUi+3 Jobi+2 VOMS extensions VOMS Server Submission Credential access information Config. file Short term X509 proxy MyProxy Medium term X509 proxy BOINC Project Owner Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  10. Bridging EGEE to BOINC and XtremWebBridge BOINC  EGEE Solution = Inside EDGeS bridge, marshalling of the BOINC Work Units into Job collections • For each (BOINC server, BOINC Project Owner, EGEE VO) triple, a separate Job Handler collects the BOINC Work Units and place them in a queue. • For each (BOINC Project Owner, EGEE VO) pair, a separate EGEE plugin : • Retrieves a short term X509 Proxy for the BOINC Project Owner from a MyProxy server, and VOMS extensions from a VOMS server, • Periodically processes new Work Units found in the queue : • It converts each Work Unit into an EGEE Job, • In order to reduce the usage of the EGEE WMS, it uses Collection possibili-ties of EGEE to submit many Jobs in one request described using JDL. Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  11. Bridging EGEE to BOINC and XtremWebBridge XtremWeb  EGEE EGEE Pushes Pilot job VOMS Server gLite WMS Computing Element Gives Pilot Job Status Submits mono-user Pilot Job with X509 proxy Gives Pilot Job Status Mono-user Pilot Job XtremWeb Bridge User Job X509 proxy Requests User Jobs Sends User Jobs with X509 proxy Requests only 1 User Job X509 proxy with VOMS extensions Manages User Job status Sends 1 User Job with same X509 proxy Submits User Job with X509 proxy XtremWeb Server Sends back Job Status and Results XtremWeb User Sends back results directly Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  12. Bridging EGEE to BOINC and XtremWebBridge XtremWeb  EGEE Solution = XtremWeb bridge : Gliding with a mono-user Pilot Job • A XtremWeb User submits to the XtremWeb server his User Job with a X509 proxy. • At the request of the XtremWeb bridge, the XtremWeb server sends him the User Job with the X509 proxy. • The XtremWeb bridge submits to a gLite WMS a mono-user Pilot Job with this X509 proxy (job description in a JDL). • The gLite WMS pushes the Pilot Job to a Computing Element, which executes it. • The mono-user Pilot Job requests 1 User Job from the XtremWeb server, and stops itself if it receives none. • The XtremWeb server verifies that the requested User Job has a X509 proxy, and sends the User Job and the X509 proxy to the Pilot Job. • The Pilot Job verifies that the received X509 proxy is the same as its own X509 proxy, and executes the User Job. • At the end of the User Job, the Pilot Job sends the Job results directly to the XtremWeb server, then stops itself. Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  13. Bridging EGEE to BOINC and XtremWebBridge EGEE  BOINC EGEE BOINC Server gLite WMS EGEE BDII LCG-CE for EDGeS EDGeS3G bridge Informationprovider Reports resources and performance Queue Manager& Job DB Sends output Logs events Submits Job Pushes job GRAM JobManagerfor EDGeS EGEE LB Logs events Generic Job WS Handler Adds job Watches job Gets output EGEE VOMS Watches BOINC plugin (DC-API) Checks EXE X509 proxywith VOMS extensions EDGeS Application Repository BOINC Computing Resource Gets EXE BOINC Service EGEE User Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  14. Bridging EGEE to BOINC and XtremWebBridge EGEE  BOINC Solution = Installation of a LCG-CE sending the EGEE Jobs to the EDGeS bridge marshalling them into BOINC Works Units • Publish information to the BDII according to GLUE 1.3 • EGEE producer • New GRAM jobmanager • Gets job information from wrapper • Checks if exe is validated in the EDGeS application repository (GEMLCA) • Checks if exe is supported by attached BOINC • Gets files from WMS • Adds job to 3G bridge job Database • Polls status of jobs in 3G bridge job Database • Gets results from 3G bridge and uploads to LB • BOINC plugin (DC-API) • Use DC-API to generate BOINC WUs • Jobs are read from the 3G bridge DB • 3G DB entries are updated on events • The plugin has already been implemented for the CancerGrid system Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  15. Bridging EGEE to BOINC and XtremWebBridge EGEE  XtremWeb Solution Inside a LCG-CE, installation of a GRAM jobmanager to marshal the EGEE Jobs into XtremWeb Jobs. EGEE gLite WMS EGEE BDII LCG-CE for XtremWeb Informationprovider Reports resources and performance Sends output Logs events Submits Job Pushes job XtremWebServer GRAM JobManager forXtremWeb EGEE LB Adds job Watches job Logs events Gets output EGEE VOMS Watches Checks EXE X509 proxy with VOMS extensions EDGeS Application Repository XtremWeb Computing Resource Gets EXE EGEE User Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  16. Bridging EGEE to BOINC and XtremWebArchitecture of the EDGeS 3G Bridge Control path WU  Job EDGeS 3G Bridge Generic interface above grid plugins Storage for received jobs Work Unit Grid plugin (submit jobs, update status, get output, ...)‏ JobDatabase User Job with X509 proxy BOINC Server BOINC Handlers EGEE Plugins EGEE gLite WMS WU Job EGEE Queue Manager Control path Job  WU Job Handler Interface GridHandler Interface gLite WMS BOINC Plugins (DC-API) Scheduler EGEE Handler Job BOINC Server LCG-CE for EDGeS WU Handlerfor received jobs XtremWeb Plugins XtremWeb Server Generic handlerfor received jobs Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  17. Bridging EGEE to BOINC and XtremWebDesktop Grid Production Infrastructure EGEE Job EGEE User gLite WMS EDGeS 3G bridge LCG-CE for BOINC LCG-CE for XtremWeb BOINC plugin (DC-API) BOINC Desktop Grids XtremWeb Desktop Grids Public DG SZDG 30.000 PCs Local DGUoW Grid 1.500 PCs Public DG AlmereGrid 3.000 PCs Local DG IN2P3 Grid 200 PCs Public DG Extremadura Grid 70.000 PCs Public DG EGEE@home Planned 10.000 PCs Public DGEGEE  XtremWeb 1.000 PCs Public DG INRIA Grid 300 PCs Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

  18. Bridging EGEE to BOINC and XtremWebOGF standards used for future interoperability • GLUE 2.0 in order topublish information to the BDII :Needs implementation by gLite. • BES to receive Job submissions : For example from GridSphere Portal. • BES to submit Jobs :Needs availability of CREAM CE. • JSDL to describe Jobs :Needs implementation by gLite. Potentially : • AUTHZ for Authentication / Authorization • UR, RUS for Job logging and accounting • ByteIO, SRM, GridFTP, DMI to manage data transfers • ACS for the GEMLCA application repository • SAGA, DRMAA for the methodology of application development Authors : Z. Balaton, G. Caillat, Z. Farkas, G. Fedak, G. Gombas, P. Kacsuk, A. Kornafeld, J. Kovacs, H. He, O. Lodygensky, A. Marosi, E. Urbah

More Related