Unix Programming Environment Part 3-2 Using Shell 2002/09/25 Draft Version

Unix Programming Environment Part 3-2 Using Shell 2002/09/25 Draft Version 2002/10/01 Rev 1.0

Agenda • Preamble – Shell Operations • 1. Basic Shell Features • 2. Shell Scripts • 3. Reading the textbook ( Chapter 3 )

Preamble (1) – Shell Operations • Reads its input from a file, from a string supplied as an argument to the `-c' invocation option, or from the user's terminal. • Breaks the input into words and operators, obeying the quoting rules. These tokens are separated by metacharacters. Alias expansion is performed by this step. • Parses the tokens into simple and compound commands. • Performs the various shell expansions, breaking the expanded tokens into lists of filenames (“Filename Expansion”) and commands and arguments. • Performs any necessary redirections and removes the redirection operators and their operands from the argument list. • Executes the command. • Simple commands expansion, command search, execution environment, exit status and signals • Optionally waits for the command to complete and collects its exit status.

Preamble (2) • => • Commands ( Simple and Compound ) • Metacharacters and Quoting Rule • Shell Expansions • I/O Redirection • Searching the command • Command Execution Environment

1. Basic Shell Features - Commands( 1 ) • 1. Shell commands • 1.1 Control operator • A word that performs a control function. It is a newline or one of the following: `||', `&&', `&', `;', `;;', `|', `(', or `)'. • 1.2 Simple Commands • sequence of words separated by blanks, terminated by one of the shell's control operators. The first word generally specifies a command to be executed. • its exit status is provided by the POSIX.1 waitpid function, or 128+n if the command was terminated by signal n. • 1.3 Pipelines • [time [-p]] [!] command1 [| command2 ...]

1. Basic Shell Features - Commands ( 3 ) • 1.4 Lists of Commands • A list is a sequence of one or more pipelines separated by one of the operators `;', `&', `&&', or `||', and optionally terminated by one of `;', `&', or a newline. • `&&' and `||' have equal precedence, followed by `;' and `&', which have equal precedence. • Commands separated by a `;' are executed sequentially; the shell waits for each command to terminate in turn. The return status is the exit status of the last command executed. • command && command2 :command2 is executed if, and only if, command returns an exit status of zero. • command || command2:command2 is executed if, and only if, command returns a non-zero exit status. • $ file=lll.txt; [ -e $file ] || touch $file

1. Basic Shell Features - Commands ( 2 ) • 1.5 Grouping Commands • two ways to group a list of commands to be executed as a unit. When commands are grouped, redirections may be applied to the entire command list. • ( list ) • Placing a list of commands between parentheses causes a subshell to be created, and each of the commands in list to be executed in that subshell. Since the list is executed in a subshell, variable assignments do not remain in effect after the subshell completes. • { list; } Placing a list of commands between curly braces causes the list to be executed in the current shell context. No subshell is created. The semicolon (or newline) following list is required. • Examples from the textbook: ( p52 ) • (date; who) • (date; who) | tee save | wc

1. Basic Shell Features - commands( 4 ) • 1.6 other commands • Looping Constructs : Shell commands for iterative action • Conditional Constructs: Shell commands for conditional execution • We will discuss the above commands in the Part-4( B-Shell Programming ).

1.2 Basic Shell Features - Metachar • 2. Metacharacters in B-Shell • Book, p54, table3-1

1.3 Basic Shell Features - Quoting • 3. Quoting : • Quoting is used to remove the special meaning of certain characters or words to the shell. • Quoting can be used to disable special treatment for special characters, to prevent reserved words from being recognized as such, and to prevent parameter expansion. • Escape Character • A non-quoted backslash `\' preserves the literal value of the next character that follows, with the exception of newline. • If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). • Single Quotes • Enclosing characters in single quotes preserves the literal value of each character within the quotes. • Double Quotes • Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of `$', ``', and `\'. • The characters `$' and ``' retain their special meaning within double quotes. • The backslash retains its special meaning only when followed by one of the following characters: `$', ``', `"', `\', or newline. • A double quote may be quoted within double quotes by preceding it with a backslash. • The special parameters `*' and `@' have special meaning when in double quotes. (Shell Parameter Expansion )

1.4 Shell Expansion (1) • 4.1 Overview • Expansion is performed on the command line after it has been split into tokens. There are six kinds of expansion performed: • Tilde Expansion: Expansion of the ~ character. • Shell Parameter Expansion: How Shell expands variables to their values. • Command Substitution: Using the output of a command as an argument. • Arithmetic Expansion: How to use arithmetic in shell expansions. • Word/Field Splitting: How the results of expansion are split into separate arguments. • Filename Expansion: A shorthand for specifying filenames matching patterns. • Only for Bash: • Brace Expansion: Expansion of expressions within braces. • Process Substitution • After all expansions, “quote removal” is performed.

1.4 Shell Expansion (2) • 4.2 Tilde Expansion ( ~ ) • If a word begins with an unquoted tilde character (`~'), all of the characters up to the first unquoted slash (or all characters, if there is no unquoted slash) are considered a tilde-prefix. • Expansion Rule: • If none of the characters in the tilde-prefix are quoted, then tilde-prefix is treated as a possible login name. • the tilde-prefix will be replaced by a pathname of the home directory associated with this login name . • If this login name is the null string, the tilde is replaced with the value of the HOME shell variable. • If the login name is invalid, or the tilde expansion fails, the word is left unchanged. • Example: echo ~xzy; echo ~; echo ~kill; echo $HOME

1.4 Shell Expansion (3) • 4.3 Parameter Expansion • Format: ${expression} • The simplest format: ${parameter} or $parameter • for positional parameters with more than one digit • If the parameter name or symbol is not enclosed in braces, the expansion will use the longest valid name: $Fx • Formats with modifiers: • ${parameter:-word} • Use Default Values. • ${parameter:=word} • Assign Default Values • ${parameter:?[word]} • Indicate Error if Null or Unset • ${parameter:+word} • Use Alternative Value.

1.4 Shell Expansion (4) • 4.4 Command Substitution (Book, p62 ) • Command substitution allows the output of a command to be substituted in place of the command name itself, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting . • Two formats: • $(command) • (backquoted version): `command` • When using the $(command) form, all characters between the parentheses make up the command; none are treated specially. • If the substitution appears within double quotes, word splitting and filename expansion are not performed on the results. • Example: • echo $( < filename ); echo $( cat filename ) • $ arch=$(uname –m); echo $arch; arch=`uname –m`; echo $arch • PS1="`whoami`@`hostname`$ “; PS1="$(whoami)@`hostname`$"

1.4 Shell Expansion (5) • 4.5 Arithmetic Expansion • evaluating an arithmetic expression and substituting its value • Format: $((expression)) • The shell will expand all tokens in the expression for parameter expansion, command substitution and quote removal. • If the expression is invalid, the expansion will fail and the shell will write a message to standard error indicating the failure. • Example: • $ x=100; x=$(($x – 1)); echo $x • $ x=100; x=`expr $x - 1`; echo $x • $ x=100; x=$((`expr $x - 1` - 1)); echo $x

1.4 Shell Expansion (6) • 4.6 Field Splitting / Word Splitting • The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did NOT occur within double quotes for word splitting. • Treating each character of $IFS as a delimiter. The default of $IFS is <space><tab><newline>. If IFS is unset, then the shell use its default. • If the value of IFS is null, no word splitting occurs. • if no expansion occurs, no splitting is performed.

1.4 Shell Expansion (7) • 4.7 Brace Expansion ( only in Bash ) • Brace expansion is a mechanism by which arbitrary strings may be generated. This mechanism is similar to filename expansion, but the file names generated need not exist. • Patterns to be brace expanded take the form of an optional preamble, followed by a series of comma-separated strings between a pair of braces. The preamble is prepended to each string contained within the braces. • Example: • $ echo a{b,c,d}e • mkdir /usr/local/src/upe/{old,new,dist,bugs}

1.4 Shell Expansion (8) • 4.8 Filename/Pathname Expansion • Pattern Matching: *, ?, [] • The slash character and the period character in a pathname: MUST be explicitly matched • Specified patterns are matched against existing filenames and pathnames, as appropriate. • 4.9 Misc • Process Substitution: named pipe • History Expansion: “history”, “!n”

Examples (1) # 1. "Command Expansion" release=$(cat /etc/redhat-release) arch=$(uname -m) echo "the current system: $arch - $release" echo # 2. "Command Expansion" # o Expecting a nicely ordered directory listing # here.However, what you get is messy : # the newlines disappeared. dir_listing=`ls -l` echo $dir_listing # unquoted, Remember "Word Splitting" here # 3. keep the newlines echo "$dir_listing" # quoted, no word splitting here

Examples (2) ## ${parameter}, $parameter ## o value of the variable parameter. ## o In certain contexts, only the less ambiguous # ${parameter} form works. ## o concatenating variables with strings. username="${username}-on-`hostname`“ echo "username: $username“ ## ${parameter-default}, ${parameter:-default} ## o If parameter not set, use default. username=${username-`whoami`} echo "username: $username"

1.5 I/O Redirection ( 1 ) • 5.1 • files for the current shell execution environment or for any command • Redirection operators can be used with numbers representing file descriptors. • [n]<word; [n]>word ; [n]>|word; [n]>>word • [n]>&word: Duplicating an Output File Descriptor • 5.2 Here Documents / Here-document • This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input for a command. • Format: <<[-]word here-document delimiter

1.5 I/O Redirection ( 2 ) • Notes: • If any character in word is quoted, the delimiter is formed by performing quote removal on word, and the here-document lines will not be expanded. Otherwise, the delimiter is the word itself. • If no characters in word are quoted, all lines of the here-document will be expanded for parameter expansion, command substitution and arithmetic expansion. • If the redirection symbol is <<-, all leading tab characters will be stripped from input lines and the line containing the trailing delimiter. • Example: $ x=1 $ cat <<eof1; cat <<eof2 > Hi, I am ${x}0 > eof1 > hi, nice to meet you, ${x}. > eof2

1.6 How does Shell search commands? • If the command name contains no slashes, the shell attempts to locate it. If there exists a shell function by that name, that function is invoked. • If the name does not match a function, the shell searches for it in the list of shell builtins. If a match is found, that builtin is invoked. • If the name is neither a shell function nor a builtin, and contains no slashes, Bash searches each element of $PATH for a directory containing an executable file by that name. If the search is unsuccessful, the shell prints an error message and returns an exit status of 127. • Using one hash table to accelerate searching • If the search is successful, or if the command name contains one or more slashes, the shell executes the named program in a separate execution environment. Argument 0 is set to the name given, and the remaining arguments to the command are set to the arguments supplied, if any. • If this execution fails because the file is not in executable format, and the file is not a directory, it is assumed to be a shell script and the shell executes it. • No “magic number” in the shell script

1.7 Command Execution Environment • Command Execution Environment • When a simple command other than a builtin or shell function is to be executed, it is invoked in a separate execution environment. Unless otherwise noted, the values are inherited from the shell. • open files inherited by the shell at invocation, as modified by redirections supplied to the exec builtin • the current working directory • the file creation mode mask( by “umask” or inherited ) • current traps set by trap • shell parameters/variables( including the environment variable ) • shell functions various process Ids( $$, $PPID, etc ) • Options( enabled by shopt, by set, command-line arguments ) • shell aliases defined with alias, etc

1.8 Misc. • 1. Alias Builtins • “alias” • Example: $ alias ll='ls –l’; alias lsd=‘ls -d */’ • 2. The directory stack • The directory stack is a list of recently-visited directories. • The pushd builtin adds directories to the stack as it changes the current directory, and the popd builtin removes specified directories from the stack and changes the current directory to the directory removed. • The dirs builtin displays the contents of the directory stack. • Command Line Editing • Using the readline library: gdb, etc • Default bound key sequence: • “C-a”: Move to the start of the line. “C-e”: Move to the end of the line. • “M-f”: forward a word, “M-b”: move back a word

2.1 Creating New Commands • The classical UNIX philosophy: breaking complex projects into simpler subtasks, chaining together components and utilities. • Executable binary and Shell Scripts • Shell Scripts • A shell script is a text file containing shell commands. • Shell Scripts: supported by the kernel, magic number( “#!/bin/sh” ) • using the chmod command to turn on the execute bit, “fork” a new process to exec “sh” which interprets the script( execution environment) • Format: • filename arguments ( sh filename arguments )

2.2 Shell Parameters & Variables (1) • Shell Parameters – Book, p59 • 1. Positional Parameters: The shell's command-line arguments. • Positional parameter N may be referenced as ${N}. • The positional parameters are temporarily replaced when a shell function is executed. • When a positional parameter consisting of more than a single digit is expanded, it must be enclosed in braces. • $ echo "Parameter #10 is ${10}“ • 1 2 3 4 5 6 7 8 9 0 10 • Parameter #10 is 0 • $0: the command’s name

2.2 Shell Parameters & Variables (2) • 2. Special Parameters: Parameters with special meanings. • The shell treats several parameters specially. These parameters may only be referenced; assignment to them is not allowed. • *: "$*" is equivalent to "$1c$2c...", where c is the first character of the value of the IFS variable. • @: " $@" is equivalent to "$1" "$2" .... When there are no positional parameters, "$@" and $@ expand to nothing (i.e., they are removed). • #: Expands to the number of positional parameters in decimal. • ?: Expands to the exit status of the most recently executed foreground pipeline. • $: Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the invoking shell, not the subshell. • $ echo $$; ( echo $$)

2.2 Shell Parameters & Variables (3) • 3. Shell Variables Revisited: • Variables other than “environment variables” can not be inherited. • Using “.” to run scripts • Change the current shell’s execution environment • The scripts can be unexecutable. ( temporarily redirecting the current shell’s standard input to the script file ) • $ var=vaule command args • Change the subshell’s execution environment • the textbook: p65

3. Textbook ( 1 ) ( Chapter 3 ) • P51-53, 3.1 • ; & newline • ( date; who ) • tee • P54, 3-2 • Table3-1 • Quoting: ‘’ \ “” • P58, 3-3 • “SubShell”: execution environment • P59, 3-4 • Positional parameters: ${100} • P61: grep “$*” /usr/you/lib/phonebook

3. Textbook ( 2 ) ( Chapter 3 ) • P62, 3-5 • Command Expansion: • $( command ) • P63, 3-6 • “Shell Execution Environment” & Process • Environment Variables: export • P66, 3-7 • Here document • P69, 3-9

Unix Programming Environment Part 3-2 Using Shell 2002/09/25 Draft Version