SZTAKI & International Desktop Grid Federation : status report

1. SZTAKI & International DesktopGridFederation: status report József Kovács (MTA SZTAKI)jozsef.kovacs@sztaki.mta.hu http://idgf-sp.eu IDGF-SP is supported by the FP7 Capacities Programme under contract nr RI-312297

2. International Desktop Grid Federation: short introduction • Legal form: Dutch foundation • Main vehicle for dissemination during the lifetime of the EC funded projects and for continuation after the projects • Federation is open for organisations and individuals in research and industry • A member organisation with chapters and working groups • Focal point for DCI/Desktop operators and application developers • Webportal: discussion fora, events and workshops, support tools, etc. • Provide assistance and facilities to launch new joint actions, initiatives, projects

3. IDGF-SPInternationalDesktop Grid Federation - Support Project Involve and engagein long-term significantly more citizens and new communitiesin the • volunteer and • private (campus-wide or enterprise) Distributed Computing Infrastructures by supporting the (1) rapid creation, (2) efficient operation, and (3) dynamic expansion of this type of DCIs for e-Science. Coordinateand synchronise the dissemination and supportactivitiesof major European stakeholders of volunteer and Desktop Grids with focus on the International Desktop Grid Federation. Supported by: EC in the FP7 Project acronym: IDGF-SP Contract n°: RI-312297 Project type: CSA-SA Start date: 01/11/2012 Duration: 24 months Funding from the EC: 860.000 € Total funded effort in person-month: 136 Web site: http://idgf-sp.eu Coordinator: Dr. Robert Lovas email: rlovas@sztaki.hu

4. SUMMARY: software components of maintained by SZTAKI • SZDG boinc package: debian variant of BOINC supported by our tools • 3GBridge: remote job submission, converting incoming jobs to WUs • DC-API: c library for creating master-worker applications • Genwrapper: porting legacy application to BOINC, shell interpreter • GBAC: virtualisation under BOINC client, based on virtualbox • Metajob: submission of large number of jobs to 3Gbridge in one turn • Cloud handling by 3G Bridge: dynamic on-demand cloud bursting • Monitoring by 3G Bridge: information collection on job throughput • EGI: gLite CREAM mCE: automatic job transfer from gLite to BOINC • EGI: Munin and Nagios:EGI compliant monitoring • gUSE: web-based graphical portal for executing jobs and workflows

5. Documentation site: doc.desktopgrid.hu

6. Recent (ongoing) activities and plans (I.) • Supporting the latest BOINC release by our tools (DC-API, 3GBridge, …) • plan is to put DC-API into the BOINC trunk • testing has been successfully done • need David to integrate our staff into the BOINC code repo • Improving GBAC • handling of stdout & stderr of application has been improved • returning the log of the internal GBAC component for debugging purposes • automatic image extraction on the client side (image size 700MB -> 100MB) • short term plan is to support image selection (discussing what is the easiest/most efficient way to implement it…) • Improving 3GBridge • 3GBridge supports REST interface, implementation finished • http://doc.desktopgrid.hu/doku.php?id=manual:restif • consumed CPU hours, credits can now be queried (required for EGI accounting & ELTE)

7. Recent (ongoing) activities and plans (II.) • Volunteer Storage • Using random linear coding in distributed storage • See presentation from Adam Visegradi • New applications on our public, volunteer projects • BIOVEL community applications on EDGeS@home • Biome-BGC ecosystem modeling is the estimation of behavior of terrestrial ecological systems (forests, grasslands, agricultural areas) as accurately as possible. • New mathematical application will appear on SZTAKI Desktop Grid these days • Riemannzetaresearch project: http://www.riemann-siegel.com/ • Locating large values of zeta functions on the critical lines • New user community: DIRAC • The DIRAC system has been recently integrated with BOINC through GBAC • The GBAC VM image has been adjusted to the needs of DIRAC • DIRAC jobs are currently arriving through gLite • Testing finished, deployment on EDGeS@home will happen these days

8. Recent (ongoing) activities and plans (III.) • Cloud scenarios • We created cloud images to support automatic setup of BOINC server with clients • To ease the deployment • To perform dynamic, on-demand computing infrastructure for scientific users • To encourage scientific users to utilise BOINC • We are creating images for each scenario defined at doc.desktopgrid.hu • Image with BOINC client • Needs a proper configuration to force the client to continuously execute WUsin the cloud • Not so easy in the new scheduling algorithm, recompilation is needed • Adjusting related parameters hardwired in the client_state.h would help… • #define WORK_FETCH_PERIOD 60 • #define WF_MIN_BACKOFF_INTERVAL 600 • #define WF_MAX_BACKOFF_INTERVAL 86400 • #define WF_DEFER_INTERVAL 300

9. European Grid Infrastructure(short overview) • European • Over 35 countries • Grid • Secure sharing of IT resources • Infrastructure • Computers (clusters): 372,612 CPUs • Data: 180 PB disk • Applications: 1.67 million job/day (thousands of scientists) • …. andbeyond!! • Technologies to • Federate resources • Access resources Coordinator: EGI.eu, Amsterdam

10. Proposed EGI DG-VT Project • General project info • Start (planned) : September-October 2013 • Duration : 6 months • Leader: MTA SZTAKI from the Hungarian NGI • “EGI-InSPIRE is ideally placed to join together the new Distributed Computing Infrastructures (DCIs) such as clouds, supercomputing networks and desktop grids, for the benefit of user communities within the European Research Area.” --- EGI-InSPIRE Project fact sheet • Motivation • Despite the on-going work and the benefits of the integrated Desktop Grid DCI resources available for EGI users, only a part of the EGI community is awareof the latest Desktop Grid related achievements, and a fraction of the EGI users and infrastructure operators take their advantages in everyday practice. • In order to fill this gap; a part of the targeted objectives are technical ones but there is stronger focus on the human aspects – training, networking and support activities. • See list of Virtual Teams at https://wiki.egi.eu/wiki/Overview_of_Virtual_Team_projects

11. Proposed EGI DG-VT Project • Promote and train the Desktop Grid related technologies in the EGI communities • Utilize the available bridged Desktop Grid resources by more VOs • Increase the number of heavily used EGI applications on the integrated (Desktop Grid) infrastructure with focus on widespread tools/solutions/approaches • Improve documentation, e.g. Road maps, training materials, manuals • Complete the final remaining steps for full integration concerning e.g. dashboard • Find joint EGI champion(s) - IDGF ambassador(s)

12. Current status of integration into the EGI infrastructure https://wiki.egi.eu/wiki/Desktop_Grids_integration

13. EGI related activities and plans • EGI activated its efforts in full integration of the Desktop Grid systems/projects to the EGI infrastructure and in promoting Desktop Grid among the EGI scientific users • EGI plans to put manpower behind as well • Plans: • EDGeS@home will be renamed to EGI@home • Technical discussion is needed, how to do that? Share your experiences if any… • Screensavers for the EGI applications • Use the cooperation with BIOVEL and DIRAC as a positive example • More dissemination (newsletters, leaflets) to notify scientific users about DG resources

14. Thank you for your attention… Presenter: Jozsef Kovacs (jozsef.kovacs@sztaki.mta.hu) Project website: http://idgf-sp.eu http://desktopgridfederation.eu http://www.lpds.sztaki.hu Acknowledgements: • The presented work is supported by the FP7 Capacities Programme under contract nr RI-312297 (IDGF-SP).