Customizable command line tools for Grids

1. GridTools: Customizable command line tools for Grids Ian Kelley + Gabrielle Allen Max Planck Institute for Gravitational Physics Golm, Germany ikelley@aei.mpg.de

2. Introduction • Simple command line tools (in Perl) for testing and performing operations across TestBeds. • Motivation: • Working with 26 machines on the SC2001 testbed • Tools to help us get our physics users onto the Grid • Playground for easily testing different scenarios before building them into portals/applications. • Have been useful for us, so put them together and wrote some documentation. • See also TeraGrid pages: • http://www.ncsa.uiuc.edu/~jbasney/teragrid-setup-test.html

3. TestBeds • What do we mean by “TestBed”? • My definition of a TestBed: • “a collection of machines with some sort of coordinated infrastructure, that is used for a common purpose or by a specific group of users • We want to develop, deploy and test portal and application software • Ultimately: want real “users” to view TestBed as a single resource • For me: • SC2001 (GGF Apps) TestBed • GridLab TestBed • AEI Relativity Group production machines • My personal TestBed

4. SC2001 TestBed • 26 Machines • Very heterogeneous • All sites worked to build towards a common setup (GRAM, GSI, Cactus, Portal, GIIS) • At SC2001 showed a Cactus simulation dynamically spawning individual analysis tasks to all machines • http://www.aei.mpg.de/~allen/TestBedWeb/

5. NumRel Production TestBed • This is what we really want! • For physicists to do physics! • Hard work! Blue Horizon Lemieux Psi Seaborg Globus Titan Origin Platinum sr8000 Los Lobos

6. (Some) TestBed Gripes … • Software deployment not yet standard/stable • Information not easy-to-find or up-to-date • Different security/account policies (firewalls!!) • Priorities mean things not always fixed quickly. • Hard to get a global view of current state. • Have trouble keeping track of changes • Not everything works as expected  • Basically we need to work in a “research-like” environment, but the more we use it, the more “production-like” it will become …

7. What We Want To Do • Run different tests (gsi, gram, etc) on our TestBed to verify that things are working correctly. • Easily get up-to-date global views of our testbeds. • Log files for tracking history, stability, etc. • Easily add and configure machines and tests. • Construct and test more complex scenarios for applications • Something that our end-users can also use!

8. Higher Level Scenarios • For example for our portal/applications we want to test feasibility/usefulness etc of • Remote code assembly and compilation • Repositories of executables • Things specific for Cactus: parameter files, thornlists • Data description archiving, selection, transfer • Visualisation • Design of user-orientated interfaces • User customisations • Collaborative/Group issues • Simulation announcing/steering/tracking • These also require work on the applications !!

9. GridTools Aims • Give a wrapper around Globus tools that enables scripting capability to perform multiple tasks. • Provide additional functionality such as a pseudo database for storing machine and configuration specific information. • Modularization of functionality to allow for easy development of more complex programs.

10. What You Get • Basic scripts: • TestAuth • TestResources • Report making • CreateTEXT • CreateHTML • CreateMAP • Other stuff • A Library: • GridTools.pm • A Pseudo-database • grid.dat • (all the stuff we really want to get from e.g. MDS)

11. TestAuth Output

12. Current Tests for TestResources • Authorize to Globus Gatekeeper • Simple GRAM job submission • Using GSIFTP to copy files • Using GSISCP to copy files • Testing GSISSH in batchmode • Simple job run using GASS server • Simple MPI job run using GASS server • Using machine specific predefined RSLs to execute a simple job • Very simple to add new tests.

13. TestResources Output

14. TestResources Output

15. Extensibility • GridTools.pm, a Perl module, contains many common functions that allow you to easily write additional scripts or modify the existing ones. • Such as execution of commands via fork() or using timeouts • Reading of machine configuration • User (text based) interfaces • Could implement other useful functionality • Timing how long things take to complete • More advanced monitoring • How often do different services go down on different machines • Querying of information servers to update local database, or visa-versa • GridTools can be extended to perform more complicated tasks. • Such as real job submission • Using RSL templates and compilation specific information • Distribution or aggregation of files and processes

16. Conclusion • GridTools can help you to run tests on a group of computers to provide you with a general overview of the status of your TestBed. • Can be extended to include more complicated tasks such as job distribution and compilation. • Obtain from CVS: • cvs –d :pserver:cvs_anon@cvs.aei.mpg.de:/numrelcvs login • password: anon • cvs –d :pserver:cvs_anon@cvs.aei.mpg.de:/numrelcvs co GridTools • Contact me for help/comments: ikelley@aei.mpg.de.