GROWL Scripts and Certificates

1. GROWL Scripts and Certificates John Kewley Grid Technology Group E-Science Centre CCLRC Daresbury Laboratory j.kewley@dl.ac.uk

2. 3 Barriers Three main three barriers that newcomers find when using the Grid for the first time: • Setting up the client-side middleware • Handling of certificates • Job submission in the presence of firewalls

3. Client Middleware: Problems • Typically need to be root to install (according to documentation) • Software must be downloaded from various locations • There are many choices for type of installation (too many options?)

4. Installation GROWL scripts provide a simple way of installing Grid middleware on your client Linux machine: Advantages: • Don't need to be a privileged user • Will download client middleware packages for your system (if supported by VDT) • Will build client middleware (if not) • Minimal setup/configuration • About 10–15 mins (if all goes well !)

5. VDT The Virtual Data Toolkit (VDT) is an easy to install and configure ensemble of grid middleware http://vdt.cs.wisc.edu GROWL Scripts installs the pre-WS globus client from VDT including gsi-enabled OpenSSH

6. Installing Grid Client using GROWL Scripts • Download GROWL Scripts $ cd $ # possibly tell wget about http_proxy $ # export http_proxy=http://wwwcache.dl.ac.uk:8080 $ wget http://www.growl.org.uk/GrowlScripts.tar.gz • Install into home directory $ tar -zxvf GrowlScripts.tar.gz • Build VDT client (a software distribution that includes globus) $ cd Growl $ make VDT

7. Additional Setup Since you will need an amended $PATH and other environment variables setting up for whenever you run GROWL Scripts, you should add the appropriate line to .bash_profile, .login, .profile, .cshrc or whatever other file your shell uses for this purpose. For sh: . ~/Growl/setup.sh For bash: source ~/Growl/setup.sh For cshand tcsh and similar: source ~/Growl/setup.csh

8. GROWL Scripts: Contents VDT client installation of globus and MyProxy • grid-proxy-init, grid-proxy-info • globus-job-submit, globus-job-run • gsissh, gsiscp, openssl • myproxy-init, myproxy-info, myproxy-logon Certificate helper scripts • mk-cert • growl-info, growl-login, growl-logout GROWL wrapper scripts • growl-submit, growl-status, growl-get-output, • growl-sh, growl-cp, growl-mkdir, growl-rm,, • growl-pwd, growl-which, growl-get-jobmanager • growl-queue

9. 3 Barriers Three main three barriers that newcomers find when using the Grid for the first time: • Setting up the client-side middleware • Handling of certificates • Job submission in the presence of firewalls

10. $ openssl pkcs12 -clcerts \ -nokeys -in usercred.p12 \ -out usercert.pem <Pass1> $ openssl pkcs12 -nocerts \ -in usercred.p12 \ -out userkey.pem <Pass1> <Pass2> <Pass2> [confirm] $ chmod 444 usercert.pem $ chmod 400 userkey.pem $ mv userkey.pem ~/.globus $ mv usercert.pem ~/.globus $ chmod 700 ~/.globus $ mk-cert mykey.p12 <Pass1> [<Pass2>] mk-cert

11. Advantages • growl-login wraps grid-proxy-init, myproxy-init and myproxy-logon • growl-logout wraps grid-proxy-destroy and myproxy-destroy • growl-info wraps grid-cert-info, myproxy-info and grid-proxy-info

12. Some use cases for proxies • Generation of a proxy credential from a grid certificate • Uploading a proxy credential to a MyProxy server for use from other clients or portals • Retrieving a proxy credential without a grid certificate (using myproxy)

13. JK JK 12h 365d growl-login Client $ growl-login <Grid Pass phrase> $ grid-proxy-init <Grid Pass phrase> Grid Resource

14. JK JK > 3h 365d growl-login Client $ growl-login $ Grid Resource

15. Scenarios for proxies • Generation of a proxy credential from a grid certificate • Uploading a proxy credential to a myproxy server for use from other clients or portals • Retrieving a proxy credential without a grid certificate (using myproxy)

16. JK JK 12h 365d JK 7d growl-login -m Client $ myproxy-init <Grid Pass Phrase> <MyProxy Password> <MyProxy Password> $ myproxy-logon <MyProxy Password> $ myproxy-init -L <Grid Pass Phrase> <MyProxy Password> <MyProxy Password> $ growl-login -m <Grid Pass Phrase> <MyProxy Password> Grid Resource MyProxy Server

17. Scenarios for proxies • Generation of a proxy credential from a grid certificate • Uploading a proxy credential to a myproxy server for use from other clients or portals • Retrieving a proxy credential without a grid certificate (using myproxy)

18. JK JK 12h 365d JK > 1d growl-login Client $ growl-login <MyProxy Password> $ myproxy-logon <MyProxy Password> Grid Resource MyProxy Server

19. JK JK > 3h 365d JK 7d growl-login Client $ growl-login $ Grid Resource MyProxy Server

20. Advantages • Symmetry growl-login vs growl-logout c.f.myproxy-logon vs grid-proxy-destroy • Generally less passwords to type • Single command automagically "does the right thing™ " • Does nothing if no need • Utilises myproxy server if credential present and valid • Uses existing proxy credential if still valid

21. growl-info • Combines certificate info into one package • Reports on only certs and credentials it finds • Issues warnings • Notes when other certs are present (but are not the ones that growl-login would use)

22. More • VOMS integration • Simpler Certificate application and renewal

23. 3 Barriers Three main three barriers that newcomers find when using the Grid for the first time: • Setting up the client-side middleware • Handling of certificates • Job submission in the presence of firewalls

24. GROWL job submission • Help with transparency - user shouldn't really need to know • Machine's jobmanager • Home directory location • Location in your path of executable • Firewall problems minimised

25. Running a grid job (1) $ growl-submit dl1.nw-grid.ac.uk/jobmanager-sgehostname https://dl1.nw-grid.ac.uk:64010/792/116475/ $ growl-status https://dl1.nw-grid.ac.uk:64010/792/116475/ PENDING $ growl-status https://dl1.nw-grid.ac.uk:64010/792/116475/ DONE $ growl-get-output https://dl1.nw-grid.ac.uk:64010/792/116475/ comp023.nw-grid.ac.uk $ growl-submit -c dl1.nw-grid.ac.uk/jobmanager-sge hostname $ growl-status PENDING $ growl-status DONE $ growl-get-output comp021.nw-grid.ac.uk

26. Globus + Firewalls Grid Resource Client globus-job-submit jobmanager globus-job-get_result Results gsiscp sshd gsissh /GSI-SSHTerm

27. GROWL + Firewalls Client Grid Resource growl-submit jobmanager globus-job-get-output growl-get-output (using gsissh) sshd

28. Advantages growl-submit: • use of "-c" option to cache JobID string. • uses growl-which to get full path of executable, ensuring it is in your path • uses growl-get-jobmanager to obtain default parallel queue, rather than defaulting to jobmanager-fork Not currently working growl-get-output: • uses gsissh to do remote retrieval, avoiding client firewall problem

29. Remote filestore manipulation Equivalents of many of the standard unix command tools are provided for remote filestore manipulation. growl-ls : contents of directory growl-mkdir : (sub)directory creation growl-rm : file removal growl-mv : renaming/moving files growl-which : finds executable in your path growl-pwd : prints your home directory on the grid resource growl-sh :gsissh wrapper (using default ports) growl-cp : remote file copying, including "3rd party" An additional parameter (the grid resource) is required

30. Remote file copying using growl-cp growl-cp can be used to stage and retrieve files. The syntax follows that of scp. It can also be used for "3rd party" file transfers For 3rd party transfers to work, there has to be a route through all firewalls between the two remote resources in one direction or the other $ growl-cp my_input_file.txt dl1.nw-grid.ac.uk:. $ growl-cp dl1.nw-grid.ac.uk:my_output.txt . $ growl-cp lv1.nw-grid.ac.uk:my_file.txt dl1.nw-grid.ac.uk:.

31. growl-cp (1) Client Grid Resources B A

32. growl-cp (2) Client Grid Resources B A

33. growl-cp (3) Client Grid Resources B A

34. Usage patterns • Easy way to build VDT • As above + certificate scripts • As above + use of job submission features

35. Summary • Useful as an easy way to build VDT • Simpler job submission: • less need be known about Grid resources • less firewall pain for retrieving data • Firewall-aware 3rd party file copying http://www.growl.org.uk/