1 / 27

Providing uniform and standardised access to European grid middleware with SAGA

Providing uniform and standardised access to European grid middleware with SAGA. Steve Fisher, Shantenu Jha*, Andre Merzky , O Weidner, A Wilson http://saga.cct.lsu.edu EGEE-III INFSO-RI-222667. SAGA: In a nutshell. There exists a lack of programmatic approaches that:

ken
Download Presentation

Providing uniform and standardised access to European grid middleware with SAGA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Providing uniform and standardised access to European grid middleware with SAGA Steve Fisher, Shantenu Jha*, Andre Merzky, O Weidner, A Wilson http://saga.cct.lsu.edu EGEE-III INFSO-RI-222667

  2. SAGA: In a nutshell • There exists a lack of programmatic approaches that: • Provide general-purpose, basic &common grid functionality for applications and thus hide underlying complexity, varying semantics.. • The building blocks upon which to construct “consistent” higher-levels of functionality and abstractions • Meets the need for a Broad Spectrum of Application: • Simple scripts, Gateways, Smart Applications and Production Grade Tooling, Workflow… • Simple, integrated, stable, uniform and high-level interface • Simple and Stable: 80:20 restricted scope and Standard • Integrated: Similar semantics & style across • Uniform: Same interface for different distributed systems • UMD: Components of gLite, ARC, UNICORE & globus

  3. SAGA: In a thousand words..

  4. gLite CREAM Overview • CREAM = Computing Resource Execution And Management • Lightweight service for job management operation at the Computing Element (CE) level • CREAM can be used via: • gLite Workload Management System • the ICE service • a generic client (Java/C++ + command line interfaces available) • The Simple API for Grid Applications (SAGA).

  5. gLite CREAM SAGA Adaptor • SAGA provides a prototype job adaptor to access gLite CREAM CEs • Implemented with the CREAM C++ API • Currently provides the following features: • job description (saga-jd to JDL mapping) • job control (run, submit, suspend, cancel, etc...) • listing of jobs associated with a CE • re-connection to running jobs • Encapsulates (hides) proxy delegation

  6. SAGA/CREAM Python Example

  7. SAGA/CREAM C++ Example

  8. SAGA/CREAM Getting Started • The adaptor code is available via SVN:https://svn.cct.lsu.edu/repos/saga-adaptors/glite/ • Due to the limitations of the CREAM/gLite build-system the adaptor only supports platforms that are supported by gLite (Scientific Linux, etc…) • But will be release with glite in near future • Please send comments and/or questions to: oweidner@cct.lsu.edu

  9. Movie time…

  10. SAGA BES Adaptor • SAGA BES adaptor based on bes++ • C reference implementation for HPC-BP by Chris Smith (Platform) et.al. • has been used for BES-HPCBP interop • uses gSoap • provides ’besclient’ and ’besserver’ • code is somewhat messy, as usual with gsoap based code

  11. SAGA BES Adaptor (cont.) • C++ wrapper for bes++ client lib was added • simplifies adaptor code to (job::run) void job_cpi_impl::sync_run (saga::impl::void_t & ret) { if ( state_ != saga::job::New ) { SAGA_ADAPTOR_THROW ("can run only 'New' jobs", saga::IncorrectState); } job_epr_ = bes_.run (jsdl_); jobid_ = "[" + endpoint_url_.get_string () + "]-[" + job_epr_->str + "]"; { adaptor_data_typeadata (this); // scoped lock state_ = adata->get_saga_state (bes_.get_state (job_epr_)); } } • svn co https://svn.cct.lsu.edu/repos/saga-adaptors/ogf/trunk • license: GPL (due to gSoap)

  12. SAGA example submit to BES SAGA BES Adaptor (cont.) #include <saga/saga.hpp> int main () { saga::sessions; saga::contextc; c.set_attribute (saga::attributes::context_username, “ogf”); c.set_attribute (saga::attributes::context_userpass, “ogf”); s.add_context (c); saga::job::servicejs (s, ep.url); saga::job::jobj = js.run_job_job (“/bin/sleep 10”); while ( j.get_state () == saga::job::Running ) { cout << "Running" << endl; ::sleep (1); } if (j.get_state () == saga::job::Done ) cout << "Done" << endl; else cout << "Failed" << endl; return j.get_state (); }

  13. SAGA BES Adaptor (cont.) • funtional: • endpoint verification • certificate and username/pass auth • SAGA job description -> JSDL • JSDL -> SAGA job description • job submission • BES job state -> SAGA state • terminate job • not yet implemented • list backend jobs • reconnect to job based on job-id

  14. SAGA BES Adaptor (cont.) • Interop Tests: • bes++: besserver with SAGA backend ;-) • fully functional • Unicore: https://zam1161v01.zam.kfa-juelich.de:8002/ • fully functional • GridSam: https://gridsam-endpoint.oerc.ox.ac.uk:18443/ • demo certificate expired • ARC: http://knowarc1.grid.niif.hu:50000/arex • demo CA expired • Planned Tests: • NAREGI, CREAM, Genesis-II, PBS-Pro, MS HPC Server • Please provide us with interop endpoints! :-)

  15. GANGA OVERVIEW

  16. SAGA-GANGA Integration

  17. DIANE INTEGRATION Diane without SAGA Diane with SAGA DIANE is an execution manager with support for pilot-jobs + worker agents (IDEAS Redux)

  18. Lattice-QCD Applications on heterogeneous resources Federating resources! EGEE Conference (Apr’10) Payload distribution (Not in this demo: cloud resources, additional Grid infrastructures…) Master Application-aware (and resource-aware) scheduling Ganga/SAGA (to *) Ganga/SAGA (to TeraGrid) Ganga/gLite Agents scheduling Heterogeneous resources allocation (Ganga + Ganga/SAGA)

  19. RENKEI Project Aims Osaka Univ. Tsukuba Univ. Middleware-independent service & application KEK Service & Applications Svc Apps Apps SAGA HEP Library RNS Yet Another FC service based on OGF standard Python Binding C++ Interface SAGA framework SAGA-Engine Adpt Adpt Adpt SAGA adaptors SRB iRODS NAREGI gLite Cloud LRMS LSF/PBS/SGE/… This activity is funded by MEXT as a part of RENKEI project which develops seamless linkage of resources in the Grids and the local one for e-Science.

  20. BigJob: Infrastructure Independent Pilot-Job

  21. BigJob: Infrastructure Independent Pilot-Job (Each sub-job is a MPI-based MD)

  22. Service Discovery • This API (GFD.144) identifies URLs of services • Hides the GLUE-I ↔ GLUE-II transition from the user • saga-sd --sf "Site IN ('INFN-CNAF', 'RAL-LCG2')" --sa • Return details of all services you are authorized to use at INFN-CNAF and RAL-LCG2 • gLite adapter being added to UI and WN patches this week • NAREGI just starting work on an adapter • Hope to see it at next OGF in Chicago

  23. Information System Navigator • ISN exposes the schema to allow navigation over any ER style schema • gLite adapter supports GLUE-I now. GLUE-II will be tested when data are available. • Available as experimental API in LSU SAGA implementation with CLI and gLite adapter • GUI portal to SD and ISN being tested. Will be made generally available soon.

  24. SAGA: Available Adaptors • Job Adaptors • Fork (localhost), SSH, Condor, Globus GRAM2, OMII GridSAM, Amazon EC2, Platform LSF • File Adaptors • Local FS, GlobusGridFTP, Hadoop Distributed Filesystem (HDFS),CloudStore KFS, OpenCloud Sector-Sphere • Replica Adaptors • PostgreSQL/SQLite3, Globus RLS • Advert Adaptors • PostgreSQL/SQLite3, Hadoop H-Base, Hypertable • Other Adaptors • Default RPC / Stream / SD

  25. SAGA-based Tools and ProjectsAdvantage of Standards • JSAGA from IN2P3 (Lyon) • http://grid.in2p3.fr/jsaga/index.html • gLite adaptors exist • JAVASAGA (Amsterdam) • Has a wide range of adaptors • JAVASAGA gets released by gLite (next few weeks) • NAREGI/KEK (Active) • http://www.ogf.org/OGF27/materials/1767/OGF27_SAGA_KEK.pdf • DEISA/DESHL • http://www.fz-juelich.de/nic-series/volume38/pringle.pdf ) • http://deisa-jra7.forge.nesc.ac.uk/ and http://www.ogf.org/OGF19/materials/501/SAGA-DEISA.ppt • XtreemOS • http://saga.cct.lsu.edu/index.php?option=com_content&task=view&id=95&Itemid=174

  26. Summary.. • Power of well-defined, agreed upon interfaces • Tools, Gateways and Applications • Advantages across levels (close to the middleware, upto direct application level), and scales (single EGEE/EGI site, to largest TG resources) • SAGA provides application-level interoperability across different middleware and distributed resource types • Interoperability first step towards effective utilisationof DCI • IDEAS • http://www.cct.lsu.edu/~sjha/select_publications

  27. Acknowledgements SAGA Team and DPA Team and the UK-EPSRC (UK EPSRC: DPA, OMII-UK , OMII-UK PAL), NSF (HPCOPS, Cybertools) and LA-BOR People: SAGA D&D: Hartmut Kaiser, Ole Weidner, Andre Merzky, Andre Luckow, Joohyun Kim, Lukasz Lacinski, JoãoAbecasis, Chris Miceli, Bety Rodriguez-Milla SAGA Users: Andre Luckow, Yaakoub el-Khamra, Kate Stamou, Cybertools (AbhinavThota, Jeff, N. Kim), OwainKenway Google SoC: Michael Miceli, SaurabhSehgal, MiklosErdelyi Collaborators and Contributors: Steve Fisher & Group, Sylvain Renaud (JSAGA), Go Iwai & Yoshiyuki Watase (KEK) DPA: Dan Katz, Murray Cole, Manish Parashar, Omer Rana, Jon Weissman

More Related