VEIGA: Very Easy Interface to Grid Applications (formerly known as "Styx Grid Services")

1. VEIGA: Very Easy Interface to Grid Applications(formerly known as "Styx Grid Services") Jon Blower Technical Director Reading e-Science Centre jdb@mail.nerc-essc.ac.uk http://www.resc.rdg.ac.uk

2. Overview of VEIGA • Aim is to make running applications on a Grid as easy as running local applications • Apps can be run from remote locations using the same command as for running the app locally • Simple workflows can be created using simple shell scripts • Data can be streamed directly from app to app • Can run applications on Condor and Globus resources • Secure

3. Example: GULP • General Utility Lattice Program (Julian Gale, IC) • Used extensively in mineral sciences (e.g. NERC e-Minerals project) • Takes a single input file (input.txt) describing a crystal lattice • Outputs a single file (gulp_0001.txt) with bond lengths, Helmholtz free energy etc • Having deployed GULP on a VEIGA server, it is run by clients like so: • veigarun <server> <port> gulp • Or can create simple wrapper script so that we can just type: • gulp

4. How it works • GULP is installed on the VEIGA server • VEIGA server holds an XML description of the GULP program: • <gridservice name="gulp" command="c:\gulp\gulp_win.exe"> • <inputs> • <input type="file" name="input.txt"/> • </inputs> • <outputs> • <output type="file" name="gulp_0001.txt"/> • </outputs> • </gridservice> • veigarun is a completely generic VEIGA client program and performs the following tasks: • Downloads the XML description of GULP • Uses it to parse the command line to find out what input and output files to expect (in case of GULP, these happen to have fixed names) • Uploads input file(s) to VEIGA server • Starts GULP, which runs on the VEIGA server • Redirects the standard streams (stdin, stdout, stderr) • downloads the output file(s) as they are produced (doesn't wait for finish)

5. Some Benefits • Very easy to deploy and run applications in the VEIGA environment • See http://www.resc.rdg.ac.uk/veiga • Useful when application is tied to a particular platform but you want to run it from elsewhere and pretend it's local • Can easily substitute the "Grid-enabled" GULP for your local GULP • There is a GUI for GULP that wraps the GULP executable. Can simply point this to the remote GULP instead of a local installation: no change to GUI code • Can use the remote application in place of local applications in shell scripts • Easy to create "workflows" • Can use all shell constructs such as pipes etc and combine with local apps • Can arrange for data to take most direct route between servers • Can do computational steering • Simple implementation but it works • Custom clients and server plug-ins can be created using a Java API • Documentation on this is not great yet… • WSRF bindings have been created (Andrew Harrison, Cardiff): VEIGA app modelled as WS-Resource

6. VEIGA and Condor Worker nodes • If VEIGA server is a Condor submit host, can delegate jobs to a Condor pool • Helps to load balance • Not everything runs on VEIGA server • Multiple jobs can be run with a single command: • Upload directories instead of single files: • myprog –i inputs –o outputs • Easy high-throughput computing • Demo VEIGA Server + Condor VEIGA client

7. VEIGA and Globus NGS myproxy server • VEIGA can be used as a portal to Globus resources • E.g. NGS • Users log on to VEIGA with MyProxy user/password • VEIGA retrieves proxy certificate • Helps with Globus firewall problem • VEIGA server takes pain on behalf of client • VEIGA server used as staging post for files • Could transfer files direct from client to destination but would require client to have GridFTP or similar VEIGA server VEIGA client

8. Some technical points of (possible) interest • VEIGA uses the Styx protocol for distributed systems • JStyx – Java implementation of Styx developed at ReSC • http://jstyx.sf.net • Styx is ~20 years old • Single server process handles all traffic • File upload/download, progress monitoring, control messages • Styx is a multiplexing protocol • Can tunnel traffic through SSH • Client-server connections are persistent • Servers only need one incoming port open through firewall • Clients need no incoming ports open • VEIGA only requires a JRE – includes all other dependencies • Whole system is a 3MB download

9. Security • VEIGA can be set to permit access to registered users only • Contains set of usernames/passwords. Passwords stored as secure hash, not plain text • No client certificates! (my bank does not use client certs) • All traffic can be encrypted using SSL • Server has a certificate • Or, everything can be tunnelled down SSH • Because VEIGA is a single server process with single port • Keeps some sysadmins happy • For access to GSI-protected resources, users can log into a VEIGA server with MyProxy password • Then VEIGA retrieves proxy certificate for them • VEIGA behaves like a Grid portal

10. Possible test case: GCEP • Grid for Coupled Ensemble Prediction • Run ensembles of climate simulations and get stats of results • Use resources at Reading, RAL, British Antarctic Survey (BAS), NGS • Need to share results among project partners and allow partners to submit jobs • Need a resource broker (even if very simple) • Can we do the whole thing using SSH (and VEIGA)? • No Globus, no GSI, no proxy certs (except for NGS access) • SSH very well understood and trusted by all relevant sysadmins and already installed • Shared file system with sshfs (or SRB if not) • Give me a good reason why not!

11. Conclusions • VEIGA favours simplicity over efficiency • A simple tool for simple jobs • Users already know how to use it! • Remote apps can be run just like local ones • Doesn't feel like job submission • Simple workflows are created with shell scripts • Can use VEIGA as an easy-to-use "portal" to Condor and Globus • Client interface is identical in all cases • Could use same principles to launch jobs using AHE / GridSAM • VEIGA client does not know about the back end • VEIGA as AHE client plug-in? • See http://www.resc.rdg.ac.uk/veiga • (as of 4/5/06) • Contains tutorials, links to papers etc • Current release (0.2.0) does not contain everything I've talked about today – next release (0.3.0)!