Scommands Tutorial

1. Scommands Tutorial By: Roman Olschanowsky (roman2u@sdsc.edu) Presented By: Leesa Brieger (leesa@sdsc.edu)

2. Scommands • Command line access to the SRB • Included in TeraGrid CTSS (current v. 3.4.1) • Download, then compile fromhttp://www.sdsc.edu/srb/tarfiles/SRB3_4_2client.tar (make sure you get non-encrypted client only, as well as correct version that matches server version) • For a UNIX build, in the SRB directory, do: • ./configure (--enable-gsi-auth --enable-globus- location=$GLOBUS_LOCATION) • gmake • Or login to a machine with Scommand binaries • via ssh to a *nix machine • Win32 binaries from command window

3. Scommand Features • Command line interface -> SCRIPTING • Available for all of the most popular UNIX flavors and DOS • S-commands are the most flexible and powerful of the clients • They are the fastest, and most reliable • They are multithreaded for big gains in data flow • They are great for scripts, perl wrappers, batch jobs, etc… • Installed man pages via “man [Scommand]” Ex: man Sput

4. Sinit Senv Spwd Sls Scd Sget Sput Ssh Scp Smv (logical) Sphymove (physical) Srm Smkdir Srmdir Serror Schmod Common Scommands (69 total) • Don’t forget to Sexit!

5. Scommand Notes • Shelp • Gives list of commands with brief summary • “[Scommand] <return>” gives usage info (usually) or try –h flag • Sinit – establishes a session • Senv – displays connection information • Spwd – display current working directory • Sexit – ends session

6. SRB Connection parameters • Verify/Create ~/.srb/.MdasEnv • Verify/Create ~/.srb/.srbAuthFile OR • Set environment variables SRBUSER SRBPORT AUTH_SCHEME SRBHOST MCATZONE … See http://www.sdsc.edu/srb/index.php/User_Environment

7. ~/.srb/.MdasEnv File mdasCollectionHome '/home/roman2u.teragrid' srbUser 'roman2u' mdasDomainName ‘teragrid‘ AUTH_SCHEME 'GSI_AUTH' #AUTH_SCHEME 'ENCRYPT1‘ srbHost ‘srb.sdsc.edu' srbPort ‘7321' defaultResource ‘sfs-tape-tgd‘ SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb'

8. TeraGrid and SRB SRB client is part of the TG software stack All Teragrid accounts are given an SDSC SRB Teragrid account (no password, by default; GSI authentication assumed) • SRB username is the same as your SDSC UNIX account name • Your SRB ‘domain’ is ‘teragrid’ • Register your DN string with SDSC’s grid-mapfile or request an SRB password. Instructions to do so are here: http://www.sdsc.edu/user_services/srb/account_form.html

9. Environment Variables #!/bin/sh export mdasCollectionName="/home/roman2u.teragrid" export srbHost=“srb.sdsc.edu" export srbPort=“7321" export srbUser=“roman2u" export mdasDomainName=“teragrid" export defaultResource=“sfs-tape-tgd" export AUTH_SCHEME=“GSI_AUTH“ export SERVER_DN='/C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb'

10. ~/.srb/.srbAuthFile $ Sauth Enter your current SRB password: Successfully wrote ~/.srb/.srbAuthFile .$CS2/j6A$"8

11. Other authentication methods AUTH_SCHEME: 'ENCRYPT1' – random message encrypted with your password between clients & servers. 'GSI_AUTH' - Use the Globus GSI authentication scheme. Currently the SRB demo accounts du1-du40 use password authentication (ENCRYPT1); TG demo accounts use GSI (GSI_AUTH) by default.

12. Step by Step Set up SRB for TG demo accounts (ux4522_ _) • First we need to be on a UNIX machine • ssh to tg-login.sdsc.edu • soft add +globus-4.0.1 • cacl (if you have not already generated a certificate) • grid-proxy-init • gx-map -interactive • soft add +srb-client-3.4.1-r1 • mkdir .srb • cd .srb • cp /usr/local/apps/srb-client-3.4.1-r1/doc/tg.MdasEnv .MdasEnv

13. Step by Step Edit your .MdasEnv file, using your srb_username • mdasCollectionHome '/home/srb_username.teragrid’ • mdasCollectionName '/home/srb_username.teragrid’ • srbUser ‘srb_username‘ Example .MdasEnv: mdasCollectionName '/home/ux452235.teragrid' mdasCollectionHome '/home/ux452235.teragrid' mdasDomainName 'teragrid' srbUser 'ux452235' srbHost 'srb.sdsc.edu' srbPort '7321' AUTH_SCHEME 'GSI_AUTH' defaultResource 'hpss-sdsc' SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb'

14. Step by Step SRB setup for SRB demo accounts (du1 - du40) .MdasEnv (with SRB username du_ and domain “npaci”): mdasCollectionName '/home/du40.npaci' mdasCollectionHome '/home/du40.npaci' mdasDomainName 'npaci' srbUser 'du40' srbHost 'srb.sdsc.edu' srbPort '7321' AUTH_SCHEME 'ENCRYPT1' #AUTH_SCHEME 'GSI_AUTH' #defaultResource 'hpss-sdsc' defaultResource 'sfs-disk-tgd'

15. Step by Step Some basics • Sinit -v • Sls • Smkdir test • Sls • Scd test • Spwd • SgetR • Sexit (if you do this now, repeat Sinit to continue)

16. Step by Step More basics • pwd, cd • Scd • Spwd (notice you are back in your SRB home dir) • Senv • Shelp • Serror • Smkdir /home/Test • Serror -3220

17. Step by Step Hey, how many are there? • Scd /home/Demo/SRB-Tutorial/files-2 • Sls • How many Doc.txt files are there? • Sls –l • How many are there now? • They are all the same file, the file has been replicated to different resources which is different than copied to different resources. • The number after the owner column is the replication number, followed by the physical resource.

18. Step by Step Lets try it out • Scd to go to your SRB home dir • Smkdir tutorial • Scd tutorial • Scp /home/Demo/SRB-Tutorial/comics/bc.gif . • Sls –l • There should be just one replica of bc.gif • Sreplicate –S sfs-tape-sdsc bc.gif • Sls –l • Is it replicated? • Why replicate? Fault tolerance, redundancy and faster access • SgetR will list all resources • Smv only does logical move (renaming or path change) • Sphymove will move the file physically

19. Step by Step Best practices for S-command uploads and downloads • For one big file, use Sput or Sget with –M option • For a directory with lots of small files, use –b option See http://www.sdsc.edu/srb/index.php/Scommand_Manpages for descriptions of options for the S commands.

20. Step by Step Download and Upload a directory • Scd /home/Demo/SRB-Tutorial • Sget –r comics • Scd • time Sput –r comics • time Sput –b comics comics2 • time Sput –b comics comics3 • Sls –l comics; Sls –l comics2 • time Srm –r comics2 • time Srm –rf comics3

21. Step by Step SRB Trash system • Spwd • Add /trash to the front • Sls /trash/home/roman2u.teragrid • That’s why it was so fast, it did not erase it, it moved it (logically). Great news! You can recover your data, just use Smv command to put data you want back. • /trash/home/roman2u.teragrid/comics2::2005-07-26-10.39.07 • Adds timestamp to name to avoid conflicts, and to inform you when data was “trashed”

22. Step by Step Some user defined meta data • Sufmeta • Sufmeta –Q Make = Ford • Sufmeta –c Contents Funny comics • Sufmeta Parts 4 comics/bc.gif • Sufmeta Rating “5 star” comics/bc.gif • Sufmeta comics/bc.gif

23. Step by Step Put your mark on the tutorial • vi <firstname_lastname>.txt • Contents of file “Picture of my car” • Scd /home/Demo/SRB-Tutorial/cars • Sput <firstname lastname>.txt • Sls –C <firstname lastname>.txt • Schmod r npaci groups <firstname lastname>.txt • Sls –C <firstname lastname>.txt • Sufmeta Make <make> <firstname lastname>.txt • Sufmeta Model <model> <firstname lastname>.txt • Sufmeta Year <year> <firstname lastname>.txt

24. Step by Step Copying and Replicating • Scd • Scp –S sfs-tape-sdsc comics/bc.gif . • Sls –l • Sreplicate bc.gif • Sreplicate –S sfs-tape-tgd bc.gif • Sls –l • vi comics/myComic.txt • Srsync -r comics s:comics • Sbkupsrb -r -S sfs-tape-sdsc comics • Sls –l • Why did we use Sbkupsrb instead of Sreplicate? Would have worked fine the first time, but subsequent calls will keep making more replicas.

25. Using SRB in your HPC Sget input-data Compute () gather() fork() if (child) { Sput results } // Parent continues computing while child moves partial results to storage

26. Step by Step Roger, signing off… Sexit Thanks! Questions? www.sdsc.edu/srb mailto:srb-chat@sdsc.edu mailto:srb@sdsc.edu