Using SDSC TeraGrid IA-64 Cluster

1. Using SDSC TeraGrid IA-64 Cluster July 2005 Don Frederick http://www.sdsc.edu/user_services

2. Agenda • SDSC IA-64 Cluster Hardware Summary • Logging In to SDSC TeraGrid IA-64 Cluster • ssh • gsi-ssh • User Environment – softenv • Compilers • Fortran • C/C++ • Running Jobs • Batch • Interactive • Example

3. SDSC IA-64 Cluster • CPUs • Intel Intanium II 1.5 GHz, 6 MB L3 cache • System Design • 262 dual CPU nodes • 2 GB memory per node • 2 login nodes – 8 cpus/node • 16 special data transfer nodes • Interconnect • Myrinet 2000 • OS • SUSE Linux 8.0 • Disk • 60 TB GPFS • 1.7 TB NFS

4. SDSC IA-64 Cluster • PBS Batch System, Catalina scheduler • Intel Compilers – Fortran, C,C++, version 8.0 default

5. Logging on to SDSC TeraGrid IA-64 Cluster • ssh –l username tg-login.sdsc.teragrid.org • MUST use ssh or other secure login method – gsi-ssh • File copying • scp original_fileusername@tg-login1.sdsc.teragrid.org:/to_dir/copied_file • System status • http://clumon.sdsc.edu/

6. User Environment Common TeraGrid Software Stack (CTSS) –provides common set of software across TG sites • Compilers • Utilities

7. User Environment Defined using “softenv” package – allows user and/or sysadmins to define default user environment – search paths, other Unix environment variables Complete reference: • http://www.teragrid.org/docs/softenv/ • Users can customize their environment using softenv commands and mods to the “.soft” file

8. Using softenv To add and remove the softenv settings from shell environments run the respective script from the command line: /usr/sdsc/bin/add-softenv /usr/sdsc/bin/remove-softenv

9. b Manipulating the Environment • /home/<username>/.soft • softenv • Displays symbolic software key names • soft add <package-name> • Temporary addition of package to environment • soft delete <package-name> • Temporary package removal from environment • resoft • Modify “dotfile” and apply to present environment • softdb – examine softenv database • man softenv

10. TeraGrid IA-64 Compilers • Fortran90 – • Intel efc 8.1 is default compiler • C/C++ • Intel icc 8.1 is default compiler • MPI – several “flavors” • mpich-gm – mpicc, mpif90 – default mpi compilers mpich MPI library compiled using GM (Myrinet network primitives) • mpich-vmi – mpicc, mpif90 – mpich implementation allowing cross-site MPI • mpich-g2 - mpicc, mpif90 – mpich implementation allowing cross-site MPI

11. Running Jobs - Batch Portable Batch System (PBS) is the batch software. SDSC uses the Catalina job scheduler. To run in batch, create batch job script, using PBS syntax and use PBS commands to submit job to batch queue

12. Running Jobs - Batch Example of a PBS batch script for running an MPI job, executable “a.out” on 10 nodes, 2 MPI processes/node #!/bin/csh #PBS -q dque #PBS -N my_job #PBS -l nodes=10:ppn=2 #PBS -l walltime=0:50:00 #PBS -o file.out #PBS -e file.err #PBS -V cd /work/username mpirun -v -machinefile $PBS_NODEFILE -np 20 ./a.out

13. Running Jobs – Batch (Continued) Job is submitted to PBS batch system via the qsub command: qsub script-name Where “script-name” is the job script (see previous page). – must be executable. Job status checked with qstatcommand: qstat or qstat jobname Cancel jobs in queue or running with: qcancel jobname

14. Running Interactive Interactive use is via PBS: qsub -I -V -l walltime=00:30:00 -l nodes=4:ppn=2 This request is for 4 nodes for interactive use (using 2 cpus/node) for a maximum wall-clock time of 30 minutes. Once the scheduler can honor the request, PBS responds with: “ready” and gives the node names. Once nodes are assigned, ubser can now run any interactive command. For example, to run an MPI program, parallel-test on the 4 nodes, 8 cpus: mpirun -np 8 -machinefile $PBS_NODEFILE parallel-test

15. TeraGrid IA-64 – Numerical Libraries Intel Math Kernel Library (MKL) – main numerical library, optimized for IA-64 • Basic linear algebra subprograms (BLAS) with additional sparse routines. • Fast Fourier Transforms (FFT) in 1 and 2 dimensions, complex and real. • The linear algebra package, LAPACK • A C interface to BLAS • Vector Math Library (VML) • Vector Statistical Library (VSL) • Multi-dimensional Discrete Fourier Transforms (DFTs)

16. TG IA-64 Software Software • Suse linux sles8 rc4 • Intel c++ 8.0 • Intel fortran 8.0 • Intel MKL 6.0.012 • Globus 2.4.3 • Condor-G 6.5.3 • MPICH-1.2.5.10-pre3 • Open-PBS 2.3.16 • Torque 1.1.0 • Catalina • Open SSH/SSL 3.7.1 • XCAT 1.2.0beta7 • Python w/ XML

17. TG IA-64 Software The available grid services (inside Globus) are: • GPT 2.2.9 (installer) • Globus 2.4.3 • gsi-openssh 1.9 • gsi-ncftp 3.0.3 • Condor-G (NMI 2.1 binary bundle) + gahp_server 6.4.7 rebuild • Myproxy 1.4

18. Using TG IA-64 - Example Example files are in /usr/local/sdsc/training/TG. • Log onto TG IA-64 using ssh • Copy directory contents to your home directory • Run make to create executable • Use “qsub” to submit batch job script • Use qsub to submit interactive job

19. References SDSC TeraGrid User Guide • http://www.sdsc.edu/teragrid/guide.html TeraGrid “Getting Started” Guide • http://www.teragrid.org/userinfo/index.html TeraGrid Tutorial from Supercomputing ’04: “Applications in the TeraGrid Environment” • http://teragrid.org/userinfo/docs/SC04-TeraGrid-tutorial.ppt TeraGrid Tutorial on Cross-Site Data Transfer • http://teragrid.org/userinfo/docs/crosssite_xfer-sc04.ppt