1 / 26

JSAGA: Seamless Job Submission and Monitoring for Grid Infrastructures

JSAGA is a ready-to-use software that provides a single interface for job submission and monitoring across multiple grid infrastructures and middlewares. It offers an object-oriented, high-level, and uniform interface for easy usage of various technologies. The goal is to make job submission and monitoring seamless, while still supporting the complexities of different security contexts, transfer paths, and environment variables.

fleury
Download Presentation

JSAGA: Seamless Job Submission and Monitoring for Grid Infrastructures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. FJPPL meeting Lyon, 17th of February 2010 Sylvain Reynaud

  2. SAGA specification (overview) JSAGA

  3. monitor protocol control security SAGA specification / JSAGA impl. Execution Data Security JSAGA

  4. JSAGA infrastructures heterogeneity (e.g. EGEE, OSG, DEISA) • which security context to use? • which transfer path (according to • firewalls, protocols, security…) ? • which environment variables ? • how to use commands available • on workers (wget, srmcp, Sget…) ? JDL RSL EGEE OPlast delegate files staging based on SRM input data SAGA middleware heterogeneity (e.g. gLite, Globus, Unicore) GridFTP LCG-CE LCG-CE WS-GRAM • how to use the legacy APIs ? firewall job job Goal: seamless job submission job desc. JSAGA gLite plug-ins Globus plug-ins staging graph WMS WS-GRAM JSAGA

  5. Ready to use software, adapted to targeted scientific field A single interface for using all grid infrastructures A single interface for using all middlewares As many interfaces as design approaches As many interfaces as used technologies end user user applications jobs collection JSAGA application developer SAGA core engine JSAGA plug-ins interfaces JSAGA plug-ins developer plug-ins JSAGA interface Both implementer & user of SAGA implementation legacy APIs JSAGA

  6. Close to application developer needs object-oriented high-level uniform interface for all the supported technologies design objectives easy to use … but << certainly not simple to implement >> (T. Kielmann) engine code = 2 x plug-ins code Close to existing middleware APIs service-oriented low-level as many interfaces as design approaches + optional interfaces design objectives easy to implement enable efficient usage of middleware APIs SAGA plug-ins interfaces JSAGA Plug-ins interfaces in JSAGA JSAGA

  7. Plug-ins interfaces JSAGA

  8. done construction planned Plug-ins interfaces - job (streams) Streaming Plug-in interfaces: direct/buffered/redirected streams used before/during/after execution set stream for interactive set stream for non- interactive get stream for interactive getInput getOutput getError SAGA user interface: getInput / getOutput Job control gatekeeper gLite-WMS wsgram unicore6 ssh local cream naregi JSAGA

  9. Several ways to monitor jobs API mode poll job status listen to notifications about job status changes API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) Plug-in implements several interfaces User see a single interface: SAGA done construction planned Plug-ins interfaces - job (monitoring) Monitoring Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs query status for individual job listen status for individual job query status for filtered jobs getState waitFor SAGA user interface: getState / waitFor Job monitoring gatekeeper gLite-WMS wsgram unicore6 ssh local cream naregi JSAGA

  10. done construction planned rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Plug-ins interfaces - data Optimizations: • Optional interfaces for plug-ins • e.g. third-party transfer, recursive find with meta-data • Cache management of • opened connections • meta-data • content • explicit via plug-in cache Logical Physical JSAGA

  11. done construction planned plug-in supported attributes /etc/grid-security/certificates CA /tmp/x509up_u_$UID jsaga-context-init command line pass phrase proxy path proxy proxy ~/.globus/user*.pem cert key p12 Plug-ins interfaces - security X509 SSH InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS Login / pwd JSAGA

  12. done construction planned Grid techno. Others gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local gatekeeper gLite-LB wsgram unicore6 cream naregi ssh local Grid techno. Others Grid techno. Others InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS JKS SSH Login / pwd X509 JSAGA plug-ins Execution Job control Job monitoring Data Logical Physical rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Security JSAGA

  13. SAGA hides most of the differences between the technologies Information on remaining differences is available for each plug-in module… on JSAGA web site Frequently Asked Questions Configuration example with command line jsaga-help supported protocols associated security context usage, default attributes usage: jsaga-help -a <ctxId>.<attr> | --config | -d <mode> | -h | -j <mode> | -s <mode> | -v [-D <ctxId>.<attr>=<value> where: -a,--attribute <ctxId>.<attr> Output the value of security context attribute --config Output the effective configuration -d,--data <mode> Information about data protocols. <mode> = service | context -D <ctxId>.<attr>=<value> Set context instance attribute (e.g. -DVOMS[0].UserVO=dteam) -h,--help Display this help and exit -j,--job <mode> Information about job services. <mode> = service | context -s,--security <mode> Information about security context instances. <mode> = usage | default | missing -v,--version Output version information and exit JSAGA plug-ins JSAGA

  14. data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical JSAGA command line interfaces • JSAGA provides command line interfaces for… • security • jsaga-context-init • jsaga-context-info • jsaga-context-destroy • execution management • jsaga-job-run • jsaga-job-status • jsaga-job-cancel • Also useful as code examples to start with SAGA JSAGA

  15. Build process fully automated download and install build tools external libraries generate source code execute test-suites unitary tests integration tests generate project web site documentation reports generate installer GUI Plug-ins minimal external dependencies e.g. gLite-UI no needed OS-independent maven 'archetype' to generate skeleton of new plug-in project validated by a middleware-independent SAGA test suite Software quality # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base=http://grid.in2p3.fr/html/Private/ https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

  16. Installer GUI JSAGA

  17. LGPL license for the core engine and for most plug-ins Optional licenses for plug-ins having external dependencies, which license is not compatible with LGPL then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the plug-in(s) Licenses JSAGA

  18. elis@ a web portal for submitting jobs to industrial and research grid infrastructures uses JSAGA to hide grid infrastructure heterogeneity contribution from many plug-ins Job: gLite-WMS, Unicore 6, Globus WS-GRAM, SSH, local Data: RByteIO, SFTP automated SAGA execution management test-suite JUX (Java Universal eXplorer) a multi-protocols file browser contribution from P.Calvat plug-ins for SRB plug-ins for iRODS JJS (Java Job Submission) a tool for running efficiently short-life jobs on EGEE contribution from P.Calvat compare overhead with JSAGA / with direct calls to Globus API scalability tests by JJS users / Contributions logical/physical plug-ins JSAGA

  19. SimExplorer a set of tools, including a workflow engine providing distributed computing facilities, for managing simulation experiments contribution from scalability tests thousands jobs on many WMS tests with long-run workflows request + test: auto-reset job service when context renewed wiki pages jCAE(Java Computer Aided Engineering) Computer Aided Engineering on distributed computing environment contribution from currently working on SSH plug-in enable controlling and monitoring a SSH job from another JVM maybe other plug-ins later… GRIA, LSF, torque ? Contributions JSAGA

  20. (not a user of JSAGA) contribution from access to their grid services deployed for D-Grid Unicore 6 Globus v2 components (Gatekeeper) v4 components (WS-GRAM) gLite Open source technologies to manage, preserve, and link digital content (upcoming) contribution from P.Y.Jallud data plug-in to browse content from Fedora Commons repository will be included in JUX Contributions JSAGA

  21. New users => new use-cases new tests bug-fix requests new wiki pages new ideas feature requests help for testing these features need support for additional technologies contribute with new plug-ins New plug-ins for job batch systems (with DRMAA) grids clouds peer-to-peer desktop grids pilot job systems for data physical/logical file protocols application level protocols for security SSO (Shibboleth, OpenSSO…) Other ideas for contributions JSAGA

  22. Conclusion • JSAGA plug-in interfaces designed to • minimize amount of code needed to support new technologies • minimize impact of API uniformity on efficiency and scalability • JSAGA plug-in implementations minimize dependencies • on external tools and libraries (e.g. gLite-UI) • on operating system (tested on ) • JSAGA is opened to external contributions JSAGA

  23. Thank you ! JSAGA

  24. Backup slides JSAGA

  25. C C' common result R1 std-error E1 R1 C' job C SMTP SRB GSIFTP GSIFTP HTTP job OPlast EGEE CA OPlast OPlast OPlast OPlast OPlast E1 job VOMS gatekeeper srb:// srm:// CC-IN2P3 lfn:// WMS gsiftp:// EGEE Globus GSIFTP gatekeeper wsgram OpenPlast OpenPlast Grid http:// tar:// Globus gatekeeper localhost World OpenPlast Transfer path depends on… • grid or site • network filtering policy • commands available on workers • services available from workers (close Storage Element, shared FS) • supported context instances • execution service • protocols supported for staging • transfer protocol • access mode (RO, WO, RW) • third-party transfer • supported data protection level • data to stage • shared by several jobs • installed on some worker nodes • file size • required data protection level JSAGA

  26. data to stage shared by several jobs installed on some worker nodes file size required data protection level grid or site network filtering policy commands available on workers services available from workers (close Storage Element, shared FS) supported context instances E E src executable input data D1 SMTP SRB GSIFTP GSIFTP HTTP R1 C' D1 job C GSIFTP job OPlast EGEE CA OPlast OPlast OPlast OPlast OPlast TAR TAR E1 job C C' C'' common Transfer path depends on… result R1 • execution service • protocols supported for staging • transfer protocol • access mode (RO, WO, RW) • third-party transfer • supported data protection level std-error E1 E src C" E sGet JSAGA

More Related