Korn Shell Script Writing & HPC

1. Korn Shell Script Writing & HPC Science & Technology Support Group High Performance Computing OSC (Ohio Supercomputer Center) 1224 Kinnear Road Columbus, OH 43212

2. Introduction What is a shell script? Script Programming Language Why use shell scripts? Korn shell scripts References The Evolution of a Script Sampling of Script Features A simple script A more versatile Script Creating Files within a Script A documented Script Script Basics Creating a shell script Executing a shell script Echoing script commands Script Variables User-defined Variables Storing a Value into a Variable Assignment Operator = Read Command Command Substitution Null Variables Special Shell Variables Positional Parameters Referencing Variables Parameter Substitution Exercises Decision-Making statements Flow Control Exit Status If-Then-Else command Conditional Tests Logical Operators Illustrative Examples Case statement Exercises Table of Contents

3. Looping Statements for Statement select Statement while Statement Until Statement Exercises Working with Integers Arithmetic Operations Operator Precedence Numerical Bases String vs Integer Variables Exercises Script I/O Shell Redirection Operators "Here-is" Files print read Quoting Exercises Communicating with Running Scripts Sending Signals How Scripts Receive Signals Script Functions Why Use Functions? Syntax & Use Auto-Loading Function Definition Location Command Precedence HPC Designed Scripts Timing Loops Debugging Options Batch Jobs Performance Tool Preprocessing Multiple Tools runs Multiple Tool Reports File Renaming Table of Contents

4. Introduction • What is a shell script? • Script Programming Language • Why use shell scripts? • Korn Shell scripts • References

5. What is a shell script? • A shell script is a collection of Unix commands entered into a file. When the file (script) is executed, the commands are executed in order. • Running a shell script is EXACTLY equivalent to interactively entering each command • For example, say a user always does the same four steps before printing out a file: making the file owner- writable, using vi to check/change the file, printing it out, and then making the file unwritable (for safety reasons) • An interactive session of a user following this procedure for the file comp.txt is shown on the next page

6. Interactive editing and printing • $ chmod u+w comp.txt • $ vi comp.txt • Here is a table summarizing OSC computers as of today. • Supercomputer # Processors Proc Speed Ideal Calc. • ------------ ----------- --------- ------------- • Itanium Cluster 124 900 MHz 223.2 GFLOPS • Cray SV-1 16 300 MHz 19.2 GFLOPS • Origin 2000 64 300 MHz 38.4 GFLOPS • IA-32 Intel 128 550 MHz 70.4 GFLOPS • Sun Fire 6800 24 900 MHz 43.2 GLOPS • "comp.txt" 11 lines, 535 characters • $ lpr -Pps1 comp.txt • $ chmod u-w comp.txt

7. Making the viprint script • To make the previous editing and printing procedure into a script, all the user has to do is type the same commands they use interactively into a file • Say the file is called viprint, it looks like this: • $ cat viprint • chmod u+w comp.txt • vi comp.txt • lpr -Pps1 comp.txt • chmod u-w comp.txt • After viprint is made executable, whenever the user wants to execute these four commands they simply type viprint as if it was a “normal” Unix command • $ viprint

8. Script Programming Language • The real power is writing shell script comes from using special shell commands and variables that essentially make your script act like a program • In this course, you will learn about the programming statements which can be used in scripts, including: • Initializing and using your own script variables. Both single valued and arrays • Performing operations with variables • Decision-making statements (if-then-else, etc) • Looping statements (for,etc) • Defining and using functions in scripts • Script I/O commands and procedures • Above bullet reads like a “table of contents” for a text on a “traditional” programming languages such as Fortran or C, although … • Script programming has less capabilities and commands

9. Why use shell scripts? • Convenience: Run a number of Unix commands by typing a single script name • Command log: Script can permanently contain a detailed set of commands with numerous options. (If interactive, complex command is gone) • Usefulness: Write a script with programming commands to give the script a number of capabilities/options • Take “full” use of what the shell has built-in to offer you • Considering a script as a program, no compiler, loader, or libraries needed to make it executable • Portability: If a script works in the shell on your machine it will work on the same shell on a different machine • Easy to debug: Small number of programming statements which are relatively simple • Easy to modify, expand, and customize: Just type in the new commands • Run-type interaction: Can send signals to running scripts

10. Korn shell scripts • Several types of Unix shells exist, each with their own script programming languages: • Bourne (sh), Born Again shell (bash), Korn shell • C shell (csh), TENEX C shell (tcsh) • Features you would expect from a well-written shell: • Convenient interactive features: command history, command-line editing, and filename completion • Powerful scripting capabilities including reliability and efficiency • This course will discuss Korn shell scripting • It has all the convenient interactive features • It is backward compatible with the Bourne shell (the first shell) • Most scripts are sh • Start-up sequence for ksh same as sh only enhanced • Unix utilities and administration commands depend on sh • It does not have the scripting bugs that the C shell has

11. Korn shell scripts • What if I already use a different shell on my machine? • Could switch to Korn shell (use the chsh command if you can) • Keep your existing shell as your interactive shell and write your scripts as Korn shell scripts (Easily done: one line needed in the script)

12. References • Internet URLs • nacphy.physics.orst.edu/rubin/melanie/node144.html (Tutorial) • gonzo.tamu.edu/csh.whynot.html (Bugs in C shell scripting) • Usenet Newsgroups • comp.unix.shell • comp.unix.questions • Anonymous ftp • athos.rutgers.edu/~patra/vi.html (shell-101.BetaA.Z) • rtfm.mit.edu:/pub/usenet/news.answers/unix-faq/faq (Unix FAQ) • OSC Unix-Series workshops (oscinfo.osc.edu) • Basic Unix • Intermediate Unix • Introduction to Perl

13. The Evolution of a Script • Sampling of Scripting Features • A Simple Script • A More Versatile Script • Creating Files within a Script • A Commented Script

14. Sampling of Scripting Features • Next few slides will act as a course overview as we take a simple script and keep extending its capabilities • The more advanced scripts will demonstrate some of the capabilities of Korn Shell scripting • Don’t worry about understanding all the details at this point. A complete explanation of the scripting commands used is given in later chapters

15. A Simple Script • In the script called triangle, a C program is compiled and then executed. The program calculates the area of the triangle and reads in the lengths of the sides from a file and outputs the result to another file. • Here is an example of a user running the script: • $ cat triangle • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o area area.c • area < sides.in > area.dat • cat area.dat • rm area • $ ls -l triangle • -rwxr--r-- 1 dje appl 108 May 3 12:09 triangle • $ cat sides.in • 8 3 7 • $ triangle • Area of Triangle is 10.392305

16. A Simple Script • Limitation of the simple triangle script • Script only works for the one area program. What if you want to run other triangle-related programs with this same script ?

17. A More Versatile Script • Scripts can take arguments on the command line when they are run. The first argument is passed to a variable with a special name: $1. We can use this feature to make the triangle script work for any program. • Here is the resulting new script in action: • $ cat triangle • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1$1.c • $1 < sides.in > $1.dat • cat $1.dat • rm $1 • $ triangle area • Area of Triangle is 10.392305 • $ triangle perim • Perimeter of Triangle is 18.000000

18. A More Versatile Script • Limitation of versatile script: • Scripts should be as independent of possible of “outside” existing files (for portability and ease of use). Since the sides.in file is so short, we should create the file within the script

19. Creating Files Within a Script • The easiest method for creating a small “internal” file is to use “Here-is” Unix redirection. After it is used, this temporary file is removed: • $ cat triangle • if [[ $2 = debug ]]; then • cc -g -o $1 $1.c • print "$1 meant to be debugged" • else • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c • cat << EOF > sides.in • 8 3 7 • EOF • $1 < sides.in > $1.dat • cat $1.dat • rm $1 sides.in • fi • $ triangle perim • Perimeter of Triangle is 18.000000

20. A Commented Script • A bad script writing style has been shown in all incarnations of the triangle script: the lack of comments. Comments are critical to another user (or you) in understanding what the script does. • The special “comment” on the first line of script actually specifies the shell to be used for the script commands. This is required for users whose interactive and script shells differ. • On the next page is the timing script just discussed but now with proper commenting

21. A Commented Script • $ cat triangle • #!/bin/ksh • # Compiling C code found in first parameter • cc -n32 -cckr -O3 -LNO -Ofast -TARG:madd=on -o $1 $1.c • # Using redirection to make input file • cat << EOF > sides.in • 8 3 7 • EOF • # Using while loop to run the executable 17 times • let i=0 • while (( i < 17 )); do • $1 < sides.in >> $1.dat • let i=i+1 • done • # Checking output • cat $1.dat • # Clean up files no longer needed • rm $1 sides.in

22. Script Basics • Creating a Shell Script • Executing a Shell Script • Echoing Script Commands

23. Creating a Shell Script • A shell script is an executable file which is executed by the shell line-by-line. It can contain the following: • UNIX commands • shell programming statements • comments • Create using editor of choice • Can include a #! construct in first line of script to override login shell • #!/bin/ksh uses Korn shell to execute script • #!/bin/csh uses C shell to execute script • Recommend always put this in every Korn shell script. Makes it clear what the file is. "file" Unix command will even identify it as a Korn Shell script • Name should be short and descriptive

24. Executing a Shell Script There are 3 ways to execute a shell script: 1."dot" method $ . scriptname 2."just the name" method $ scriptname 3.in the background $ scriptname &

25. Executing a Shell Script • Method 1 runs the command as if you typed them in on the command line • No ksh subprocess created • Note that methods 2 and 3 require: • execute permission for scriptname chmod +x scriptname • current directory (.) must be in PATH or else must use ./scriptname

26. Executing a Shell Script • Demonstration of execution methods: • $ ps • PID TTY TIME CMD • 31612 ttyq5 0:00 sh • 31631 ttyq5 0:00 ps • $ ls -l showsh • -rw-r--r-- 1 dje appl 43 Nov 25 1997 showsh • $ cat showsh • print Executing the shell script showsh • ps • $ . showsh • Executing the shell script showsh • PID TTY TIME CMD • 31612 ttyq5 0:00 sh • 31628 ttyq5 0:00 ps

27. Executing a Shell Script • $ chmod u+x showsh • $ showsh • Executing the shell script showsh • PID TTY TIME CMD • 31578 ttyq5 0:00 ps • 31612 ttyq5 0:00 sh • 31635 ttyq5 0:00 sh

28. Echoing Script commands • For understanding or debugging scripts it is often convenient to see the script commands shown on the monitor along with the output they produce. • Accomplished using the set -x command at the beginning of the file • To turn off the echoing, make the command set +x (counter-intuitive?) • Also useful in batch job files and - on occasion - interactively • On the next page a demonstration of the power of set -x is shown

29. Echoing Script commands • $ see • /homea/dje/ksh_script • boba.osc.edu:0.0 • a_opt.txt cba_opt.txt item.set.3.txt pid.txt • b_opt.txt fant4.txt item.set.4.txt showsh.txt • c_opt.txt item.intmod.txt no_opt.txt trap.txt • $ cat see • #!/bin/ksh • pwd • echo $DISPLAY • ls *.txt • $ cat see2 • #!/bin/ksh • set -x • pwd • echo $DISPLAY • ls *.txt

30. Echoing Script commands • $ see2 • + pwd • /homea/dje/ksh_script • + echo boba.osc.edu:0.0 • boba.osc.edu:0.0 • + ls a_opt.txt b_opt.txt c_opt.txt cba_opt.txt fant4.txt item.intmod.txt item.set.3.txt item.set.4.txt no_opt.txt pid.txt showsh.txt trap.txt • a_opt.txt cba_opt.txt item.set.3.txt pid.txt • b_opt.txt fant4.txt item.set.4.txt showsh.txt • c_opt.txt item.intmod.txt no_opt.txt trap.txt

31. Script Variables • User-defined Variables • Storing a Value into a Variable • The assignment operator = • The read command • Command Substitution • Null Variables • Special Shell Variables • Positional Parameters • Referencing Variables • Parameter Substitution • Exercises

32. User-defined Variables • The Korn shell includes the following types of variables: • User-defined variables • Special shell variables • User-defined variables can be initialized, used and changed from the command line or from within a shell script. • Not "declared" like in a programming language. Just type the name of the variable • Korn shell scripts use string and integer variables • A variable name can consist of the following: • letters, digits, and the underscore character • first character of a variable name must be a letter or an underscore character • A variable can be made read-only: once it is assigned a value, the value cannot be changed • "Archaic" method: readonly variable_name • Alternate method: typeset -r variable_name

33. Storing a Value into a Variable • There are several methods for variables to obtain values. • The assignment operator = • Enter the name that you have chosen for the variable followed by an equal sign and then the value that you want to store in the variable. • To use the a variable's value, put the $ operator in front of it's name. (Read the $ as "contents of") • The typeset command can also be used for variable assignment. More up-to-date procedure, typeset has many options. • To assign a value to an integer variable, the assignment command must be proceeded by the word let • On the next page, a script is shown that demonstrates all these approaches

34. Storing a Value into a Variable • $ cat assign • street=Elm • print street <-- Most common error when using variables • print $street • my_name="John Smith" • echo $my_name • neph=Chris Sellgren • print $neph • typeset card=ace; print $card • let num=57; echo $num • let num=num+3; print $num • $ assign • street • Elm • John Smith • assign2.ksh[9]: Sellgren: not found • ace • 57 • 60

35. Storing a Value into a Variable • Second method: read in a value for a variable • So far have used print and echo (archaic) commands for printing out all the strings following them and then a carriage return • There is a complimentary command read which input a value into variables. The user type in as many strings as there are variables on the read line, and then hits carriage return • On the next page is a demo script indicating how the read command works.

36. Storing a Value into a Variable • $ cat input • print Please enter a ship name • read ship • print Please enter your full name • read fname lname • print Captain $lname, welcome aboard the $ship • echo May I call you $fname? • print Enter an integer; read num • let num=2*num • echo Twice your number is $num • $ input • Please enter a ship name • Redoubt • Please enter your full name • Horatio Hornblower • Captain Hornblower, welcome aboard the Redoubt • May I call you Horatio? • Enter an integer • 27 • Twice your number is 54

37. Storing a Value into a Variable • Use command substitution. The string(s) that a Unix command returns may be put into a script variable • The following syntax is as follows: • var_name=$(UNIX command) • As with all the script commands, command substitution can be done interactively as well • On the next page is a demo script illustrating this technique

38. Storing a Value into a Variable • 136:oscb$ cat command • dir=$(pwd) • print I am in the directory $dir • weekday=$(date +%A) • print It is $weekday • files=$(ls -lt | wc -l) • print There a $files files in this directory • me=$(whoami) • print I am $me • computer=$(hostname) • print My computer is called $computer • logged_on=$(who | cut -f1 -d' ') • print These people are also on $computer: $logged_on • 137:oscb$ command • I am in the directory /home/dje • It is Tuesday • There a 277 files in this directory • I am dje • My computer is called oscb • These people are also on oscb: osu2796 opr006 paul ysu039 osu2796 jimg mmittal jimg mmittal osu2224 osu2224 srb cls038 osu1938 jeff dje

39. Null Variables A variable can be set to a null value, even if previously assigned, using any of the following methods. There should be no spaces preceding or following the equal sign. If the contents of null character variables are printed out, a blank appears. If the contents of a “nullled” integer variable is outputted, a 0 is displayed. $ name= $ name='' $ name="” $ let num= $ unset varname

40. Null Variables All variables that don't exist are assumed null unless set -o nounset is used. Then the shell will indicate an error when an undefined variable is encountered $ unset name $ set -o nounset $ print $name ksh: name: parameter not set $ set +o nounset

41. Special Shell Variables • In addition to your own user-defined variables, other variables defined by the shell for the user are called special shell variables and are most useful • These variables are set and updated automatically by the shell. Their values cannot be changed, but they may be referenced. • The special shell variables of most use are identified by the following symbols: $# $- $? $$ $! $0 $*

42. Special Shell Variables • The variable $# contains the number of arguments typed on the command line. $ cat numargs print The number of arguments is $# $ numargs The number of arguments is 0 $ numargs 1 2 3 4 5 The number of arguments is 5 $ numargs "Hello World" The number of arguments is 1

43. Special Shell Variables • The variable $- contains the shell flags (options) of the current shell. $ print $- isum $ set +u $ echo $- ism

44. Special Shell Variables • The variable $? contains the exit status of the last command. If the exit status is 0, the command ran successfully. If it is non-zero, there was a problem • $ ls file1 data account.txt $ rm file1 $ print $? 0 $ rm dataa dataa: No such file or directory $ print $? 2 • We will see exit status again. It will be used to define true and false. Also, the user can set the exit status in their script by using the exit command

45. Special Shell Variables • The variable $$ contains the Process ID (PID) of the current shell process. • $ cat pid • #!/bin/ksh • ps • print $$ • $ pid • PID TTY TIME CMD • 34738 ttyq5 0:00 ps • 34744 ttyq5 0:00 sh • 34765 ttyq5 0:00 pid • 34765 • $ . pid • PID TTY TIME CMD • 34743 ttyq5 0:00 ps • 34744 ttyq5 0:00 sh • 34744

46. Special Shell Variables • The variable $! contains the process ID number of the last command sent to the background. • $ sleeper & • [1] 29144 • $ ps • PID TTY TIME CMD • 29144 ttyq5 0:00 sleeper • 29150 ttyq5 0:00 sh • 29160 ttyq5 0:00 ps • 29167 ttyq5 0:00 sleep • $ compress moby_dick & • [2] 29136 • $ print $! • 29136

47. Special Shell Variables • The variable $0 contains the name of the script currently being execute • $cat old_script • print The name of this script is $0 • $old_script • The name of this script is old_script • $mv old_script new_script • $new_script • The name of this script is new_script • The variable $* contains all the arguments on the command line. $ cat args print The arguments are: $* $ args bob dave The arguments are: bob dave

48. Positional Parameters • For a script, the command line arguments are also called positional parameters. Each positional parameters refers to one individual argument on the command line. The ten positional parameters available to the script writer are referenced as follows: $1 $2 $3 $4 $5 $6 $7 $8 $9 • The parameter $1 contains the first argument, $2 the second argument, and so on. Consider the following script: $ cat parms print Arg 1 is: $1 print Arg 2 is: $2 print Arg 3 is: $3 print Arg 4 is: $4 print Arg 5 is: $5

49. Positional Parameters $ parms 2001 A Space Odyssey Arg 1 is: 2001 Arg 2 is: A Arg 3 is: Space Arg 4 is: Odyssey Arg 5 is: $ parms "Space, the final frontier" Arg 1 is: Space, the final frontier Arg 2 is: Arg 3 is: Arg 4 is: Arg 5 is:

50. Positional Parameters Use the set command to change positional parameters. It replaces existing positional parameters with new values $cat newpos print starting args are $* print number of args is $# print arg 1 is $1 print arg 2 is $2 set NCC 1701 Enterprise print new args are $* print number of args is $# print arg 1 is $1 print arg 2 is $2 print arg 3 is $3