1 / 31

EDGI E uropean D esktop G rid I nitiative

EDGI E uropean D esktop G rid I nitiative. Advanced EDGI technologies Jozsef Kovacs Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI) Budapest, Hungary kovacs.jozsef@sztaki.mta.hu

jace
Download Presentation

EDGI E uropean D esktop G rid I nitiative

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EDGIEuropeanDesktopGridInitiative Advanced EDGI technologies Jozsef Kovacs Computer and Automation Research Institute,Hungarian Academy of Sciences (MTA SZTAKI) Budapest, Hungary kovacs.jozsef@sztaki.mta.hu EDGI is supported by the FP7 Capacities Programme under contract nr RI-261556 2010-04-27

  2. Contents • MetaJob Submission of job collection in EDGI • GBAC Virtualization on the BOINC resources • Attic A P2P file system supporting the EDGI infrastructure • SpeQuloS Dynamic extension of BOINC with Clouds

  3. Software architecture developed by EDGI UNICORE MCE AR attic monitor UNICORE grid upload Attic FS UI submit DG client attic DG site ARC grid UserIF 3GBridge select cloud AR attic DGPro-ject submit BridgeIF Volunteer/CampusResources monitor DG client ARC MCE CREAM MCE attic AR AR attic attic monitor monitor gLite grid Monitor submit UI inspect Service Grid Desktop Grid EDGI bridge components

  4. A high level scenario for grid users • Step 1: Select an application you want to execute on Grids • Use any from the EDGI repository • Bring your own application • Step 2: EDGI project performs the necessary preparations for you and as a result the application appears in the EDGI application repository • application porting • registering the application in the AR and in several DGs • infrastructure setup for your access (by the operators of the connecting grids) • Step 3: Go to our EDGI AR and collect info for your submission • Step 4: Prepare your input files, create and submit jdl • Step 5: Query the status and download the outputs

  5. Step 3 (cont.): Collect informationfrom the EDGI AR (files)

  6. Step 4, 5: submit jdl and get results • Create your my.jdl (as usual): Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "pools.txt", OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; • Submit, Status query, Output downloading can be performed by the well-known gLite commands: glite-wms-job-submit, glite-wms-job-status, glite-wms-job-logging-info, glite-wms-job-output, etc.

  7. METAJOB Submission of job collection in EDGI

  8. Software architecture developed by EDGI UNICORE MCE AR attic monitor UNICORE grid upload Attic FS UI submit Single job DG client attic Unfolding DG site ARC grid UserIF 3GBridge select Huge number of jobs cloud AR attic MetaJob as a single job DGPro-ject Single job submit BridgeIF Volunteer/CampusResources monitor DG client ARC MCE CREAM MCE attic AR AR attic attic monitor monitor gLite grid Monitor submit UI inspect

  9. Step 4b: create and submit jdl • Create your jdl: Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "pools.txt", "_3gb-metajob-dsp-10000" }; OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; • Submit: glite-wms-job-submit-oid edgi-metajob-10000.jdl MetaJob definition as Extra input file

  10. MetaJob: input filesand metajob definition • Upload your individual input files to web server: • http://somewhere.com/pools1.txt • … • http://somewhere.com/pools10000.txt • Create the description of your metajob: %Comment pools1.txt Arguments = "-i 0 -n pools.txt -f 22 -p 723“ Input = pools.txt=http://somewhere.com/pools1.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue … … %Comment pools10000.txt Input = pools.txt=http://somewhere.com/pools10000.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue

  11. MetaJob: Query the status and logging Location of the status description

  12. MetaJob: Query the detailed status Metajob handling rules Actual status of jobs Location of the mapping

  13. MetaJob: Download the results • Download result as usual: • glite-wms-job-output -i id --dir outputs • ./outputs/cost.txt • Extract it: • tar zxvf cost.txt • ./outputs/<subjobid1>/cost.txt • ./outputs/<subjobid2>/cost.txt • … • ./outputs/<subjobid10000>/cost.txt • See the mapping between your individual job definition and the jobids (which gives the name of directories storing the output files of your app) one subjob id

  14. GBAC Virtualization on the BOINC resources

  15. Virtualisation on the worker nodes… • Current situation • Only registered applications can be executed on BOINC • Applications need to be ported to BOINC • Applications need to be validated • Aim of virtualisation/sandboxing: • To execute application under BOINC without registration • To execute an application on a BOINC client machine in a secure way • To do this without any porting effort in a very similar way as an application is submitted and executed on Linux, for example. • Execute applications that are not registered at the BOINC project and that were not validated for BOINC • Majority of Desktop Grid resources are Windows based (68.9%) - however majority of scientific applications run on Linux...

  16. Virtualisation by GBAC for Desktop Grids (BOINC) (SZTAKI) Desktop Grid Server Worker machine (Windows) client Worker machine client 1 GBAC 2 4 ….. Linux OS APP 3

  17. Application and Inputs Overview A User submits her application and inputs via gLite. 3G Bridge detects that the application is “legacy” (not BOINC native) and redirects the binaries and inputs to the GBAC native BOINC application at EDGIDemo. Clients (who have VirtualBox installed) will download the BOINC native GBAC application with the submitted application and its inputs. GBAC starts a Linux Virtual Machine (using VirtualBox). GBAC copies the applications and inputs into the Virtual Machine. The application is executed in the Linux VM. The result is fetched from the VM by GBAC. The VM is shut down and discarded. GBAC finishes and the result is returned to EDGIDemo from the Client. The results are returned to gLite through 3G Bridge from EDGIDemo. EDGIDemo BOINC project … 3G Bridge BOINC Client BOINC Client … GBAC GBAC Application and Inputs Application and Inputs

  18. Details: job execution on a virtualized DG resource The result is returned to gLite through 3G Bridge from the BOINC Server. A user submits an Application with inputs through gLite to BOINC. G-Lite Clients (with VirtualBox installed) will download the BOINC native GBAC application with the submitted application and its inputs. … GBAC starts a Linux Virtual Machine (using VirtualBox). BOINC Server GBAC copies the applications and inputs into the Virtual Machine. Desktop Grid resource The application is executed in the Linux VM. VirtualBox BOINC Client Application and inputs Application and inputs The result is fetched from the VM by GBAC. Virtual Machine GBAC The VM is shut down and discarded. GBAC finishes and the result is returned to BOINC from the Client. Output files Output files

  19. GBAC: VM Guest and requirements • Only Linux support • Licensing issues with Windows • 32 bit applications preferred • Linux image is 32 bit, later 64 bit one will be added • VM is a standard Linux system • Same requirements as for applications to be executed in a Grid • Static linking preferred, all non standard libraries must be bundled, etc.

  20. VirtualBox and BOINC Manager and Virtual Machine booting via GBAC NOTE: This screenshot of GBAC is using a special “DEBUG” mode for demonstration purposes. Normally no windows are presented at all and the VM always runs in the background (in so-called “headless mode” ).

  21. ATTIC A P2P file system supporting the EDGI infrastructure

  22. BOINC Client Application Computing resource EDGI – Data Access Project task BOINC Scheduler local storage project data Data handling under BOINC Volunteer PC Retrieve WorkUnit Web Server Input Upload Handler Output = Einstein@Home data mirroring locations

  23. DG Client Application (XtremWeb/BONIC) EDGI – Data Access Computing resource Project task Management service local storage project data Decentralization of data centers P2P Cloud Volunteer PC 3 LFN1 Input Input 1 LFN1 Project server Task 2 LFN1

  24. EDGI – Data Access Publishing Scheduler DataCenter XML message (over HTTP) DataLookupServer Publisher attic://voldemort.cs.cf.ac.uk:7000/data/<ID> Periodically query for replication requests attic://voldemort.cs.cf.ac.uk:7000/data/<ID> DC DC DC Register as replica; added to locations list

  25. EDGI – Data Access Publishing Publisher Any entity that publishes a file DataLookupServer Manages number of replicas for a new request and responding to cache requests Scheduler DataCenter XML message (over HTTP) DataLookupServer Publisher attic://voldemort.cs.cf.ac.uk:7000/data/<ID> Periodically query for replication requests attic://voldemort.cs.cf.ac.uk:7000/data/<ID> DC DC DC Scheduler Keeps track of managing number of replicas for a new request. Register as replica; added to locations list DataCenter Caches data on the network.

  26. Attic in the EDGI infrastructure UNICORE MCE AR attic monitor lfn upload UNICORE grid Attic UI DC PUB LS submit DG client attic DG site ARC grid UserIF 3GBridge select cloud AR attic DGPro-ject ARC MCE CREAM MCE submit BridgeIF Volunteer/CampusResources monitor AR AR attic attic monitor monitor DG client attic gLite grid Monitor submit UI inspect

  27. SpeQuloS Dynamic extension of BOINC with Clouds

  28. Motivation for using clouds • Desktop Grid • Variable amount of resources, volatility, unpredictability, unannounced departure • Low QoS compare to classical DCI • Tail Effect • We define QoS as a level of confidence in Bag of Task (BoT) execution. • Question: how do we provide QoS to users given the dynamism and volatility of the computing resources ? • Intrinsic approach : improve DG scheduler for QoS ability • Extrinsic approach : provide additional dedicated computing resources • Solution • Allow users to express QoS needs for their BoT • Provision resources from Cloud to satisfy these needs

  29. SpeQuloS: a middleware for QoS to Desktop Grids

  30. RI-261556WP1 version: 02 When to trigger QoS ? Estimates the completion time : • Without the tail • With the tail • With QoS Triggers QoS • If there is credits • At 90% of the BoT execution • if there is an expected benefit

  31. Thank you for your attention… Contact: Jozsef Kovacs (smith@sztaki.hu) Project website: www.edgi-project.eu Acknowledgements: • EDGI EU FP7 project (RI-261556)

More Related