1 / 13

Parameter study workflow execution support in P-GRADE portal

P-GRADE and WS-PGRADE portals supporting desktop grids and clouds Peter Kacsuk MTA SZTAKI kacsuk@sztaki.hu. Parameter study workflow execution support in P-GRADE portal. P-GRADE Portal. GEN. Generates input parameter space. SEQ. SEQ. SEQ. SEQ. Parameter sweep jobs:

felice
Download Presentation

Parameter study workflow execution support in P-GRADE portal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. P-GRADE and WS-PGRADE portals supporting desktop grids and clouds Peter KacsukMTA SZTAKIkacsuk@sztaki.hu

  2. Parameter study workflow execution support in P-GRADE portal P-GRADE Portal GEN Generates input parameter space SEQ SEQ SEQ SEQ Parameter sweep jobs: To be executed in Grids, Desktop Grids and Clouds COLL Evaluates the results of the simulation

  3. Motivation: Local DG in Univ. of Westminsterbased on BOINC (SZTAKI Desktop Grid) 6 1 2 5 4 • New Cavendish Street 576 nodes • Marylebone Campus 559 nodes • Regent Street 395 nodes • Wells Street 31 nodes • Little Tichfield Street 66 nodes • Harrow Campus 254 nodes Total: 1881 nodes 3 Lifecycle of a node: • PCs basically used by students/staff • If unused, switch to Desktop Grid mode • No more work from DG server -> shutdown (green solution) 3

  4. Generating DG applications • To port an application to a BOINC system requires three steps: • Registering the application at the BOINC server • Creating the master code running on the server • Creating the client code • In BOINC Step 2 and 3 require the modification of the original application and this is not trivial • Using the gUSE/WS-PGRADE environment: • Step 1 by the DG system administrator • Step 2 and 3 are done nearly automatically without any code modification

  5. Automatic generation of master and client code • To facilitate Step 2 and 3 SZTAKI developed the DC-API (Distributed Computing API) as part of the SZTAKI Desktop Grid (SZDG) package • DC-API can automatically generate WUs for PS jobs arriving from the workflow • However, DC-API still requires the modification of the original application to create the client code • SZTAKI has also developed Genwrapper a generic wrapper that can • eliminate the boincification of the code • automatically generates the client code without the modification of the original code

  6. gUSE architecture and usage User action, external event or time triggering WS-PGRADE File storage Workflow storage gUSEServices Workflow Engine Meta-broker LocalSubmitter Web ServiceClient EGEESubmitter Desktop GridSubmitter DatabaseClient DG server Dedicatedsite WebService WMS DBMS

  7. Connecting gUSE with DGs in CancerGrid DG Server gUSE Storage gUSE 3G Bridge Submitter BOINC Server Components BOINCclient GenWrapper forbatch execution DC-API cli WS-PGRADE (User IF) (WF repre-sentation) Jobdescr. Sche-duler Job Database (Description of Jobs:Apps, Args, I/O files) LegacyApplication Jobdescr. Dataserver Queue Manager gUSE LocalSubmitter BOINCclient Schedulingpolicy BOINCTask DB gUSE(Workflowenactor) DC-API master GenWrapper forbatch execution WU DC-API cli Batchcreation gUSE WS Submitter LegacyApplication gUSE gUSE-DG integration DesktopGrid

  8. Protein Molecule Simulation using AutoDock UoW Local DG WS-PGRADE portal

  9. P-GRADE Portal EC2, Eucalyptus PS job execution in Grids by P-GRADE gLite Grid Target Grid Plugins ARC Grid BOINC Grid 3G Bridge

  10. Get Output Submit job Check status P-GRADE portal → 3G Bridge P-GRADE Portal 3G Bridge Job DB Grid Plugin1 Tomcat HTTPD Runtime Grid Plugin2 Grid Handler Interface Queue Manager BOINC Plugin3 WSClient WSSubmitter Download Manager Cloud Pluginn Files

  11. P-Grade Portal P-GRADE portal supporting Clouds Host A Job Command • Job is submitted from P-Grade Portal to the 3G-Bridge. • 3G-Bridge submits the job to a Condor Clusterusing the DC-API Condor Plugin (Queue 1). • The cluster consists of workers running in the cloud. • The Scheduler keeps track of the number of jobs in the Condor cluster and of the number of the running Cloud Resources (workers). • If the cluster is overcommitted, the Scheduler starts new workers by submitting a job to the 3G-Bridge queue of the Cloud Plugin (Queue 2). • If the cluster under utilized, the Scheduler stops some workers (cloud resources) by cancelling some jobs in the Cloud Plugin Queue (Queue 2). 3G-Bridge Scheduler Information Queue 1 … … Queue 2 … … DC-API – Condor Plugin Cloud Plugin Condor Submitter/ Master Amazon/ Eucalyptus Cloud Interface Cloud Resource 1 (Condor Worker) Legend Cloud Resource N (Condor Worker) Cloud Resource 2 (Condor Worker) …

  12. Conclusions • By the P-GRADE portal you can submit PS jobs to • Grids • Local clusters • BOINC desktop grids • Clouds • By writing new target DG plugins you can easily add new type of DG resources • By writing new target Cloud plugins you can easily add new type of Cloud resources

  13. Thank you

More Related