1 / 14

Using ICENI to run parameter sweep applications across multiple Grid resources

Using ICENI to run parameter sweep applications across multiple Grid resources. Case Studies on Grid Applications – GGF10. Murtaza Gulamali Stephen McGough, Steven Newhouse, John Darlington London e-Science Centre Department of Computing, Imperial College London. Contents. The GENIE project

mackensie-acosta
Download Presentation

Using ICENI to run parameter sweep applications across multiple Grid resources

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using ICENI to run parameter sweep applications across multiple Grid resources Case Studies on Grid Applications – GGF10 Murtaza Gulamali Stephen McGough, Steven Newhouse, John Darlington London e-Science Centre Department of Computing, Imperial College London

  2. Contents • The GENIE project • The ICENI middleware • GENIE as an ICENI application • Summary and conclusions • Acknowledgements

  3. The GENIE projectBackground 3D atmosphere Atmospheric CO2 3D ice sheets 2D sea ice 2D land surface 3D ocean Land biogeochemistry Ocean biogeochemisty Ocean sediments Schematic diagram of model framework for GENIE. Courtesy of T. Lenton, CEH Edinburgh, UK. • Grid ENabled Integrated Earth system model. • Investigate long term changes to the Earth’s climate (i.e. global warming) by integrating numerical models of various components of the Earth system.

  4. The GENIE projectBackground • Grid ENabled Integrated Earth system model. • Investigate long term changes to the Earth’s climate (i.e. global warming) by integrating numerical models of various components of the Earth system. • Require a Grid infrastructure to: • flexibly couple together components to form a unified Earth System Model (ESM). • execute the resultant ESM efficiently and accurately. • archive and share the resultant data produced by the model. • provide a high-level open access system to allow a virtual organisation of Earth System modellers to collaborate.

  5. The GENIE projectPrevious scientific work • Investigate the vulnerability of the thermohaline circulation of the world ocean using a prototype model consisting of just 3 coupled components. • Run simulation across two different parameter ranges.  perform 31  31 = 961 individual simulations.  parameter sweep application!

  6. The GENIE projectPrevious e-scientific work • Provided Grid infrastructure to support this activity… • flocked Condor pool between three institutions. • web-portal to allow experiment management. • database management system (based on Geodise) to allow data archiving and retrieval. • Disadvantages of this infrastructure… • firewalls!… between institutions hosting Condor pools. • web-portal not very flexible… model and parameter choices hard-coded. • true resource brokering not taking place… all compute and database resources belonging to virtual organisation not utilised. • Solution: use ICENI middleware

  7. The ICENI middlewareBackground • ICe-Science Networked Infrastructure. • Developed by LeSC Grid Middleware Group. • Service oriented Grid middleware. • Represents compute, storage and software resources as services. • Services can communicate using standard protocols (eg. Jini, SOAP, JXTA). • ICENI provides an end-to-end middleware consisting of: • Grid service infrastructure • dynamic service management framework • application toolkit

  8. The ICENI middlewareApplication development in ICENI matrix source linear equation solver LU decomposition linear equation solver vector sink vector source linear equation solver Cholsky decomposition • ICENI uses a component programming model to describe Grid applications.  application development  application composition • Example: linear equation solver

  9. The ICENI middlewareApplication development in ICENI service list composition pane parameters

  10. GENIE as an ICENI applicationParameter sweep as component app. GENIE GENIE GENIE binary component binary component binary component setup component archive component splitter component collator component

  11. GENIE as an ICENI applicationExecuting over multiple resources GENIE GENIE resource launcher resource launcher binary component binary component setup component archive component Condor pool splitter component collator component Beowulf cluster

  12. GENIE as an ICENI applicationResults • Using ICENI, ran 4 GENIE parameter sweep experiments on Beowulf Cluster (using Sun Grid Engine) and Linux PC based Condor pool. • Sun Grid Engine: 481 jobs • Condor pool: 480 jobs • Total: (31  31 =) 961 jobs • Find that ICENI takes ~2 minutes to schedule and submit jobs to both high throughput job managers. • Each experiment took ~5 days to run.

  13. Summary and conclusions • Are able to execute GENIE parameter sweep experiments across multiple resources administered by members of virtual organisation. • Execution time same as before but: • Can leverage all the flexibility of a service oriented Grid middleware. • Can create ICENI Grid based on resources owned and federated by collaborators in the virtual organisation. • Don’t have to contend with firewalls… (sort of)

  14. Acknowledgements • My co-authors: • Dr. Stephen McGough, Dr. Steven Newhouse, Prof. John Darlington. • The ICENI development team: • http://www.lesc.ac.uk/iceni/ • The GENIE team: • http://www.genie.ac.uk/

More Related