1 / 46

ECaccess A portal to access ECMWF

ECaccess A portal to access ECMWF. Also known as EasyAccess Dominique Lucas. Content. ECaccess concepts telnet, ssh and ftp access X11 access ECaccess file management ECaccess job management Unattended file transfers ECaccess vs. ecbatch Tutorial. ECaccess - Concepts.

jabir
Download Presentation

ECaccess A portal to access ECMWF

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ECaccess A portal to access ECMWF Also known as EasyAccess Dominique Lucas

  2. Content • ECaccess concepts • telnet, ssh and ftp access • X11 access • ECaccess file management • ECaccess job management • Unattended file transfers • ECaccess vs. ecbatch • Tutorial

  3. ECaccess - Concepts • ECaccess provides a portal to access ECMWF archiving and computing facilities • Strict authentication via SecurID card andX509 certificates • Data integrity/confidentiality guaranteed by SSL

  4. ECaccess - Concepts • ECaccess provides: • File and job management in batch or interactive mode through an extended FTP server • File and job management through a Web browser • A secure telnet access to ECMWF • A secure X Windows access to ECMWF • All this via internet or RMDCN.

  5. ECaccess - Architecture • Multi tiers • ECaccess Client, ECaccess Gateway, ECaccess Server, ECaccess ecgate ECaccess ECcert ECcmd ECaccess Server Command Server Tools (Java (Java app) ECaccess FTP app) Gateway Client I F JSSE-SSL I N R Telnet OpenSSL-SSL R Client ECproxy T M ECios E Server Web Server E D W (C app) Browser (C app) R C A L N N L E FTP ECMWF ECaccess Server T Gate- CA MS way (OpenSSL) X MS Work- Server Ecgate ECaccess Gateway station

  6. ECaccess - Architecture • ECaccess client • Standard Telnet, FTP, ssh and HTTP/S client • Standard X Windows and FTP Server • ECaccess tools (eccert …)

  7. ECaccess - Architecture • ECaccess gateway • Entry point for all ECaccess users • Verifies ECaccess users authentication(certificates or passcodes) • Implements Telnet, FTP and HTTP/S protocols • Secure tunnels through firewalls to ECaccess Server • Either local gateway installed at your site or ecaccess.ecmwf.int for internet and msaccess.ecmwf.int for RMDCN.

  8. ECaccess – Architecture • ECaccess Server • Located at ECMWF: ecaccess.ecmwf.int or msaccess.ecmwf.int • Entry point for all ECaccess gateways • Authenticate ECaccess gateways • Provides job and file management functions • Provides functions to monitor file transfers • Provides system access functions • Keeps track of users activity

  9. ECaccess - Architecture • ECaccess ecgate • Runs the ECaccess Certificate Authority (CA) • Provides job management through NQS and LoadLeveler • Provides access to home, scratch and ECFS • Keeps job input, output and error files • Manages spool for unattended file transfers

  10. ECaccess – local gateways • Use local ECaccess gateways if installed • Internet ECaccess gateways available at: • ctbto.de, dmi.dk, knmi.nl, sma.ch, irmet.ie, smhi.se, • meteo.fr, dwd.de, cerfacs.fr, mercator.ocean.fr, • eumetsat.de, univie.ac.at, met.hu, cscs.ch, inm.es • RMDCN ECaccess Gateways available at: • metoffice.com, inm.es, meteor.gov.tr • If local ECaccess gateway not installed, then use ecaccess.ecmwf.int or msaccess.ecmwf.int

  11. ECaccess – telnet/ssh {mshost}$ telnet ecaccess.ecmwf.int … login: xyz Passcode: XXXXXX Which system: ecgate … ecgate{~xyz}:1  {mshost}$ ssh xyz@ecaccess.ecmwf.int … login: xyz Passcode: XXXXXX Which system: ecgate … ecgate1{~xyz}:1  • Where xyz is your ECMWF User ID • On local gateways, port 9023 (telnet) or 9022 (ssh) may be needed.

  12. ECaccess - ftp {mshost}$ ftp ecaccess.ecmwf.int … login: xyz Passcode: XXXXXX … ftp> ls 227 Entering Passive Mode. (193,61,196,110,135,230) 150 Opening ASCII mode data connection drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECJOBS drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECSCRATCH drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECTMP drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECFS drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECHOME 226 Transfer complete ftp> • sftp not yet available. • On local gateways, port 9021 may be needed.

  13. ECaccess - ftp Local environment Local file can be dragged and dropped into ecaccess area. ftp entry for batch access

  14. ECaccess – X11 via telnet {mshost}$ echo $DISPLAY hostname:0.0 {mshost}$ xhost +ecaccess.ecmwf.int ecaccess being added to access control list {mshost}$ telnet ecaccess.ecmwf.int … (Passcode validation required) Which host …: ecgate Which proxy … : X … ecgate{~xyz}:1  echo $DISPLAY (Something like … )ecaccess.ecmwf.int:<NN> ecgate{~xyz}:2  xcdp& • A control window showing the DISPLAY to be used will appear. This window or an X11 application should remain present to keep the X11 proxy.

  15. ECaccess – ecxterm via ssh {mshost}$ echo $DISPLAY hostname:0.0 {mshost}$ ssh –X xyz@ecaccess.ecmwf.int (N.B. Port number IS mandatory) … (Passcode validation required) Which host …: ecgate ecgate{~xyz}:1  echo $DISPLAY (Something like)ecaccess.ecmwf.int:<nn> ecgate{~xyz}:2  xcdp& • A control window will also appear. Keep it while X11 access required.

  16. ECaccess – ecxterm

  17. ECaccess – web – http://ecaccess.ecmwf.int/

  18. ECaccess tools - ECtools • Must be installed at your end on each platform for which access to ECMWF is required. • If you (still) had/have access to “old” ecbatch commands be careful about the $PATH as some commands are “ambiguous” between the two packages e.g. eccert, ecget, ecput. • Help for each command is available with “-help” option: • {mshost}$ eccert –help • ECtools are also available at ecmwf.

  19. ECaccess - eccert Command {mshost}$ eccert –verbose echost: ecaccess.ecmwf.int ecport: 443 eccert: /home/xyz/.eccert.crt Certificate request ECMWF user identifier: xyz Passcode from your SecurID card: Certificate saved (855 bytes) {mshost}$ • Certificate valid for 7 days. • Certificate only needed for file, job and transfer management, NOT required for unattended transfers (see later - ectrans). • Certificate not needed for ECMWF local use of ECtools.

  20. ECaccess environment

  21. ECaccess File Management • No support for meta-characters.

  22. ECaccess file management {mshost}$ ecls script.sh ecaccess-tools.tar.gz ecaccess.doc ecaccess {mshost}$ ecdir ecaccess 10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30 . 3194 drwxr-xr-x 47 xyz systems 4096 Mar 14 09:30 .. 12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55 client 124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38 gateway {mshost}$ ecget ecaccess-tools.tar.gz {mshost}$ ecget ecaccess-tools.tar.gz tools.tar.gz {mshost}$ ls *.tar.gz ecaccess-tools.tar.gz tools.tar.gz {mshost}$

  23. ECaccess file management {mshost}$ ecdelete ecaccess-tools.tar.gz DELE command successful {mshost}$ ecput ecaccess-tools.tar.gz {mshost}$ ecmkdir ectest MKD command successful {mshost}$ ecrmdir ectest RMD command successful {mshost}$ ecrmdir ecaccess Directory not empty {mshost}$

  24. ECaccess file management {mshost}$ export ECDOMAIN=ecfs {mshost}$ ecdir 10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30 . 3194 drwxr-xr-x 47 xyz systems 4096 Mar 14 09:30 .. 12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55 backup 124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38 doc {mshost}$ export ECDOMAIN=“ecfs[zzz]” … {mshost}$ export ECDOMAIN=echome … {mshost}$

  25. ECaccess Job and transfer management

  26. ECaccess Job Management {mshost}$ ecqls ecgate1 NQS submission on ecgate1 (INIT=23 ...) hpca LoadLeveler submission on hpca (INIT=21 ...) ... {mshost}$ {mshost}$ ecqls hpca diag Diagnostic jobs only temp Temporary small jobs < 5 Nodes inter_class Default interactive class debug Default interactive class os Operational serial/single task work ns Serial/single task work bench Benchmark class np Parallel work op Operational parallel work {mshost}$

  27. ECaccess Job Management {mshost}$ ecjreq -help Syntax: JREQ ECaccess-queue remote-script [args ...] -at - start date (yyyy-MM-dd HH:mm) -nd - no directives within the input script -tg - specify the target gateway name -tr - specifiy the access method (msuser[@destination]) -to - transfer output file when the request ends -te - transfer error file when the request ends -ti - transfer input file when the request ends -tk - keep in spool (default: deleted if transfer successful) -mu - send mail for the request to the stated address -mb - send mail when the execution/transfer begins -me - send mail when the execution/transfer ends -mf - send mail when the execution/transfer fails

  28. ECaccess Job Management {mshost}$ ecjreq ecgate script.sh –me 34850 {mshost}$ ecjreq ecgate test.sh Error opening file {mshost}$ {mshost}$ ecjput ecgate1 test.sh –me 34851 {mshost}$

  29. ECaccess Job Management {mshost}$ ecjls 4421 ecgate WAIT Nov 24 21:33 3884 hpca@hpca.ecmwf.int DONE Nov 18 11:42 3146 ecgate1 DONE Nov 15 21:33 {mshost}$ ecjls 3884 Jobid: 3884 Location: hpca@hpca.ecmwf.int Date/Time: Nov 18 11:42 Status: DONE stdout size: 221 stderr size: 219 stdin size: 241Jobid: 4421 {mshost}$

  30. ECaccess Job Management {mshost}$ ecjget o34852 {mshost}$ ls JOB* JOB-o34852 {mshost}$ ecjget o34852 job.out {mshost}$ ls –ail job.out 71196 -rw-r----- 1 xyz group 686 Mar 13 19:10 job.out {mshost}$ {mshost}$ ecjdel 34852 JDEL command successful {mshost}$ • ecjdel kills job running and removes all files.

  31. ECtrans Command • Unattended file transfers, issued from ECMWF • It is preferable if the ECaccess gateway is installed at your end. RMDCN ecgate, or hpca: ECAccess Member State Gateway:

  32. ECtrans - msuser maintenance

  33. ECtrans - msuser maintenance

  34. ECtrans - msuser maintenance

  35. ECtrans - msuser maintenance

  36. ECtrans - msuser maintenance • MS associations are specific to each ECaccess gateway.

  37. ECtrans Command – at ECMWF {ec-host}$ ectrans –help usage: ectrans [-get] -gateway name \ –remote msuser@destination \ -source name [args ...] -gateway str - target gateway name -source str - source file name -echost str - eccmd host name (default: localhost) -ecport num - eccmd port number (default: 644) -remote str - target user (default: same as ecuser) -target str - target file name (default: same a source) -verbose - verbose mode on -help - this message -get - transfer file from your site to ECMWF -status - check the status on file transfer Default values can also be set by the ECHOST, ECPORT, ECUSERor GATEWAY shell variables. {ec-host}$

  38. ECtrans Command {ecgate1}$ ectrans –gateway ecaccess.ecmwf.int \ -remote from_usl@genericFtp –source ./data.grib \ –target data.grib \ -verbose echost: ecaccess.ecmwf.int ecport: 644 ecuser: usl source: ./data.grib target: data.grib keep : false option: reject File to upload (585 bytes) 10129250216245 {ecgate1}$ • Users can maintain MS user’s associations on local gateway, via the web interface. • ectrans is available in both directions.

  39. ECaccess transfer Management {mshost/echost}$ ectreq –help Syntax: TREQ source [args ...] -remote str - target user (default: current) -gateway str - target gateway name (default: current) -target str - target file name (default: same as source) -keep - keep the request in the spool -reject - if the target file already exists (default) -append - if the target file already exists -resume - if the target file already exists -erase - if the target file already exists {mshost}$ {mshost/echost}$ ectreq job.out 10161191420729 {mshost}$

  40. ECaccess transfer Management {mshost/echost}$ ectret 10161191420729 TRET command successful {mshost}$ • $SCRATCH used for temporary file storage.

  41. ECaccess transfer Management {mshost/echost}$ ectls 10161191420729 STOP xyz@ecaccess.ecmwf.int Mar 14 15:19 10164593445344 DONE xyz@ecaccess.ecmwf.int Mar 14 15:22 10161193432443 DONE xyz@ecaccess.ecmwf.int Mar 14 15:19 {mshost}$ ectls 10161191420729 Copyid: 10161191420729 MS user: xyz Hostname: ecaccess.ecmwf.int Access: ECaccess gateway Status: STOP Error message: No target location for xyz Date/Time: Mar 14 15:19 Source: ./job.out Target: ./job.out {mshost/echost}$

  42. ECaccess vs. ECbatch

  43. ECaccess vs. ECbatch

  44. ECaccess vs. ECbatch

  45. ECaccess vs. ECbatch

  46. ECaccess pointers • http://www.ecmwf.int/services/ecaccess • User’s manual • Administrator’s manual • Registration center (for local ECaccess gateway) • Quick start – check first if local gateway available • http://ecaccess.ecmwf.int/ • telnet ecaccess.ecmwf.int • ssh [-X] xyz@ecaccess.ecmwf.int • ftp ecaccess.ecmwf.int • ectrans

More Related