GROWL Scripts: Lightweight Access to Grid Resources

1. GROWL Scripts: Lightweight Access to Grid Resources John Kewley Grid Technology Group e-Science Centre STFC Daresbury Laboratory j.kewley@dl.ac.uk UK e-Science All Hands Meeting

2. GROWL • Collaborative project between CCLRC (now STFC) Daresbury Laboratory and the Universities of Cambridge and Lancaster, funded by the JISC VRE programme. • Project Objectives: to produce a lightweight client-side Grid connection toolkit. • Project completed January 2007 UK e-Science All Hands Meeting

3. GROWL • GROWL addresses the three barriers that newcomers find when using the Grid for the first time: • Setting up the client-side middleware • Handling of certificates • Job submission in the presence of firewalls This talk looks at GROWL Scripts, GROWL Web Services takes a different approach UK e-Science All Hands Meeting

4. Client Middleware: Problems • Typically need to be root to install (according to documentation) • Software must be downloaded from various locations • There are many choices for type of installation (too many options?) UK e-Science All Hands Meeting

5. Installation • GROWL scripts provide an alternative way of installing Grid middleware on your client Linux machine to that given on the NGS website: • Advantages: • Don't need to be a privileged user • Will download client middleware packages for your system (assuming it is supported) • Minimal setup/configuration • About 10–15 mins (if all goes well !) UK e-Science All Hands Meeting

6. The Virtual Data Toolkit (VDT) is an easy to install and configure ensemble of grid middleware • http://vdt.cs.wisc.edu • GROWL Scripts installs the pre-WS globus client from VDT, as well as gsi-enabled openssl and the best known (IGTF accredited) CA certificates. UK e-Science All Hands Meeting

7. Installing Grid Client using GROWL • Download GROWL Scripts $ cd $ wget http://www.growl.org.uk/Growl.tar.gz • Install into home directory $ tar -zxvf Growl.tar.gz • Build VDT client (a software distribution that includes globus) $ cd Growl; make VDT • Before using any GROWL Scripts, bash users should source ~/Growl/setup.sh • while csh users should source ~/Growl/setup.csh UK e-Science All Hands Meeting

8. GROWL Scripts: Contents • Certificate helper scripts • mk-cert • growl-info, growl-login, growl-logout • VDT client installation of globus and MyProxy • grid-proxy-init, grid-proxy-info • globus-job-submit, globus-job-run • gsissh, gsiscp, openssl • myproxy-init, myproxy-info, myproxy-logon • GROWL wrapper scripts • growl-submit, growl-status, growl-get-output, • growl-sh, growl-cp, growl-mkdir, growl-rm, growl-mv, • growl-pwd, growl-which, growl-get-jobmanager, growl-queue UK e-Science All Hands Meeting

9. Certificate Manipulation • Hard to remember openssl commands are wrapped for you • Fewer passwords need to be entered • Correct file and directory permissions are applied UK e-Science All Hands Meeting

10. $ openssl pkcs12 –in mykey.p12 \ -clcerts –nokeys -out usercert.pem <Pass1> $ openssl pkcs12 –in mykey.p12 \ –nocerts -out userkey.pem <Pass1> <Pass2> <Pass2> [confirm] $ chmod 444 usercert.pem $ chmod 400 userkey.pem $ mv userkey.pem ~/.globus $ mv usercert.pem ~/.globus $ chmod 700 ~/.globus $ mk-cert mykey.p12 <Pass1> [<Pass2>] mk-cert UK e-Science All Hands Meeting

11. growl-login • If you need to upload your certificate to MyProxy and generate a local proxy, growl-login is provided: $ grid-proxy-init ...<grid-password> $ myproxy-init Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley Enter GRID pass phrase for this identity: <grid-password> ... Enter MyProxy pass phrase:<myproxy-pass> Verifying - Enter MyProxy pass phrase:<myproxy-pass> $ growl-login Password to protect MyProxy credential:<myproxy-pass> Enter GRID pass phrase for this id:<grid-password> UK e-Science All Hands Meeting

12. growl-info • A wrapper for grid-cert-info, grid-proxy-info and myproxy-info $ growl-info Certificate Information (including validity) -------------------------------------------- subject= /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley notBefore=Jun 15 16:10:35 2006 GMT notAfter=Jun 15 16:10:35 2007 GMT Local proxy certificate(s) -------------------------- subject : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley issuer : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley identity : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley type : Proxy draft (pre-RFC) compliant impersonation proxy strength : 512 bits path : /tmp/x509up_u13445 timeleft : 11:57:19 UK e-Science All Hands Meeting

13. GROWL job submission • Help with transparency - user shouldn't really need to know • Machine's jobmanager • Home directory location • Location in your path of executable • Firewall problems minimised UK e-Science All Hands Meeting

14. Running a grid job (1) $ growl-submit dl1.nw-grid.ac.ukhostname https://dl1.nw-grid.ac.uk:64010/792/116475/ $ growl-status https://dl1.nw-grid.ac.uk:64010/792/116475/ PENDING $ growl-status https://dl1.nw-grid.ac.uk:64010/792/116475/ DONE $ growl-get-output https://dl1.nw-grid.ac.uk:64010/792/116475/ comp023.nw-grid.ac.uk $ growl-submit -c dl1.nw-grid.ac.uk hostname $ growl-status PENDING $ growl-status DONE $ growl-get-output comp021.nw-grid.ac.uk UK e-Science All Hands Meeting

15. Globus + Firewalls Grid Resource Client globus-job-submit jobmanager globus-job-get_result Results gsiscp sshd gsissh /GSI-SSHTerm UK e-Science All Hands Meeting

16. GROWL + Firewalls Client Grid Resource growl-submit jobmanager globus-job-get-output growl-get-output (using gsissh) sshd UK e-Science All Hands Meeting

17. Advantages • growl-submit: • uses growl-get-jobmanager to obtain default parallel queue, rather than defaulting to jobmanager-fork • uses growl-which to get full path of executable, ensuring it is in your path • growl-get-output: • uses gsissh to do remote retrieval, avoiding client firewall problem UK e-Science All Hands Meeting

18. Remote filestore manipulation • Equivalents of many of the standard unix command tools are provided for remote filestore manipulation. growl-ls: contents of directory growl-mkdir: (sub)directory creation growl-rm: file removal growl-mv: renaming/moving files growl-which: finds executable in your path growl-pwd: prints your home directory on the grid resource growl-sh :gsissh wrapper (using default ports) growl-cp: remote file copying, including "3rd party" • An additional parameter (the grid resource) is required UK e-Science All Hands Meeting

19. Remote file copying using growl-cp • growl-cp can be used to stage and retrieve files. The syntax follows that of scp. It can also be used for "3rd party" file transfers • For 3rd party transfers to work, there has to be a route through all firewalls between the 2 remote resources in one direction or the other $ growl-cp my_input_file.txt dl1.nw-grid.ac.uk:. $ growl-cp dl1.nw-grid.ac.uk:my_output.txt . $ growl-cp lv1.nw-grid.ac.uk:my_file.txt dl1.nw-grid.ac.uk:. UK e-Science All Hands Meeting

20. growl-cp (1) Client Grid Resources B A UK e-Science All Hands Meeting

21. growl-cp (2) Client Grid Resources B A UK e-Science All Hands Meeting

22. growl-cp (3) Client Grid Resources B A UK e-Science All Hands Meeting

23. Usage patterns • Easy way to build VDT • As above + certificate scripts • As above + use of job submission features UK e-Science All Hands Meeting

24. Current/future work • Scripting help for a simplistic DIY meta-scheduler • Use of Java CoG-kit + GSI-SSHTERM on Windows UK e-Science All Hands Meeting

25. Summary • Useful as an easy way to build VDT • Simpler job submission: • less need be known about Grid resources • less firewall pain for retrieving data • Firewall-aware 3rd party file copying • http://www.growl.org.uk/ UK e-Science All Hands Meeting