J ózsef Kovács ( MTA SZTAKI ) jozsef.kovacs@sztaki.hu edgi-project.eu

1. The EDGI (EuropeanDesktopGridInitiative) infrastructure and its usagefor the European Grid user communities József Kovács (MTA SZTAKI)jozsef.kovacs@sztaki.hu http://edgi-project.eu Start date: 2010-06-01 Duration: 24 months EDGI is supported by the FP7 Capacities Programme under contract nr RI-261556

2. furtherdeveloped by supported by • DEGISCO • Disseminate and support EDGeS results world-wide • Green IT aspects EU FP7 projects on desktop grids:EDGeS → EDGI and DEGISCO • EDGI • ARC, Unicore, Clouds • QoS with Clouds • Data intensive apps • SG→DG direction support • EDGeS • DG ↔SG integration: • gLite → BOINC, XtremWeb • BOINC, XtremWeb → gLite • Compute intensive applications

3. Focus of EDGI

4. Vision EDGeS scope only for compute intensive applications for EGEE (gLite) EDGI scope for both compute and data intensive applications for EMI/EGI (gLite, ARC, Unicore) Extend Desktop Grids with Clouds for QoS

5. Types of Desktop Grids • Volunteer (Global, Public) DesktopGrid • Aim is to collect resources for grand-challenge scientific problems • Examples: • SETI@home, Folding@home, Shakemovie@home, LHC@home • Community World Grid, IberCivis, SZTAKI Desktop Grid • Institutional (Local, Non-public) DesktopGrid • Aim is to enable the quick, easy and inexpensive creation of grid for any community (company, university, etc.) to solve their own applications • Example: • SZTAKI Desktop Grid (SZDG) local version (used within EDGeS, EDGI, DEGISCO), University of Westminster Institutional DG

6. Institutional DGs in practice – University of Westminster as an example 6 1 2 5 4 • New Cavendish Street 576 nodes • Marylebone Campus 559 nodes • Regent Street 395 nodes • Wells Street 31 nodes • Little Tichfield Street 66 nodes • Harrow Campus 254 nodes • Total: 1881 nodes 3 • Lifecycle of a node: • PCs basically used by students/staff • If unused, switch to Desktop Grid mode • No more work from DG server -> shutdown • (green solution)

7. EDGI main areas/topics • Middleware development: gLite, Arc and Unicore bridge (to DG) • Infrastructure development: bridging services • Application porting activity: already more than 30 validated scientific applications: application repository • Cloud development: DG → Cloud bridge, using clouds as resources • Improving DGs with QoS • Supporting Data Intensive Applications • Consolidation of EDGeS results: improving scalability, repository, etc. • Supporting user communities, developers, operators, etc. • European chapter of the International Desktop Grid Federation • Dissemination/support of EDGI results world-wide through EDGI andDEGISCO communication channels

8. The EDGI infrastructure ARC MCE upload AR attic monitor ARC grid Eucalyptus/Amazon Attic FS UI DG client submit attic UserIF BridgeIF AR 3GBridge down-load cloud attic DGPro-ject Volunteer/InstitutionalResources monitor DG client CREAM MCE attic gLite grid AR attic monitor submit Monitor UI inspect • Developments have been finalised by end of March 2011 • Production system will be operational from 1st of May 2011 • UNICORE extension from Autumn 2011

9. A high level scenario for grid users • Step 1: Select an application you want to execute on Grids • Use any from the EDGI repository • Bring your own application • Step 2: EDGI project performs the necessary preparations for you and as a result the application appears in the EDGI application repository • application porting • registering the application in the AR and in several DGs • infrastructure setup for your access (by the operators of the connecting grids) • Step 3: Go to our EDGI AR and collect info for your submission • Step 4: Prepare your input files, create and submit jdl • Step 5: Query the status and download the outputs

10. Step 3: Collect informationfrom EDGI AR (authorized vos, ces)

11. Step 3 (cont.): Collect informationfrom the EDGI AR (files)

12. Step 4, 5: submit jdl and get results • Create your my.jdl (as usual): Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "pools.txt", OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; • Submit, Status query, Output downloading can be performed by the well-known gLite commands: glite-wms-job-submit, glite-wms-job-status, glite-wms-job-logging-info, glite-wms-job-output, etc.

13. Latest development in EDGIto improve scalability: MetaJob concept ARC MCE upload AR attic monitor ARC grid Eucalyptus/Amazon Attic FS UI DG client Single job submit attic MetaJob as a single job Unfolding UserIF BridgeIF AR 3GBridge down-load cloud Huge number of jobs attic DGPro-ject Single job Volunteer/InstitutionalResources monitor DG client CREAM MCE attic gLite grid AR attic monitor submit Monitor UI inspect Demonstrated at EGI UF, 12th of April, 2011 at both #10 with 10.000 jobs through gLite

14. Step 4b: create and submit jdl • Create your jdl: Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp" , "pools.txt", "_3gb-metajob-dsp-10000" }; OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; • Submit: glite-wms-job-submit-oid edgi-metajob-10000.jdl MetaJob definition as Extra input file

15. MetaJob: input filesand metajob definition • Upload your individual input files to web server: • http://somewhere.com/pools1.txt • … • http://somewhere.com/pools10000.txt • Create the description of your metajob: %Comment pools1.txt Arguments = "-i 0 -n pools.txt -f 22 -p 723“ Input = pools.txt=http://somewhere.com/pools1.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue … … %Comment pools10000.txt Input = pools.txt=http://somewhere.com/pools10000.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue

16. MetaJob: Query the status and logging Location of the status description

17. MetaJob: Query the detailed status Metajob handling rules Actual status of jobs Location of the mapping

18. MetaJob: Download the results • Download result as usual: • glite-wms-job-output -i id --dir outputs • ./outputs/cost.txt • Extract it: • tar zxvf cost.txt • ./outputs/<subjobid1>/cost.txt • ./outputs/<subjobid2>/cost.txt • … • ./outputs/<subjobid10000>/cost.txt • See the mapping between your individual job definition and the jobids (which gives the name of directories storing the output files of your app) one subjob id

19. SG/DG Applications on Production infrastructure(ported by EDGeS, EDGI and DEGISCO)

20. DesktopGrid applicationsavailable @ EGI AppDB

21. Key issue: SUSTAINABILITY • The International Desktop Grid Federation (IDGF) brings together: • Desktop Grid developers • Desktop Grid operators • Application developers • Everyone else interested in Desktop Grid computing • Open membership • http://desktopgridfederation.org

22. Members of

23. Thank you for your attention… Contacts: Peter Kacsuk (kacsuk@sztaki.hu) (coordinator of EDGI, EDGeS) Jozsef Kovacs (jozsef.kovacs@sztaki.hu) (deputy coordinator, technical leader) Websites: www.edgi-project.eu www.edges-grid.eu Acknowledgements: • EDGI EU FP7 project (RI-261556) • EDGeS EU FP7 project (INFSO-RI-211727)