Shell Programming

Shell Programming Asoc.Prof. Guntis Barzdins Asist. Girts Folkmanis

Lecture outline • Shell features • Helper utilities, introduction • Connecting utilities with shell scripting • Helper utilities in detail • Piping, advanced examples • Shell scripts as files, Internal shell commands

Shell features • We will talk about bash, there might be differences for other shells. • bash - GNU Bourne-Again Shell • Authors: Brian Fox and Chet Ramey • Free Software Foundation • Popular in different distributions • Tip: To find your current shell, type following command $ echo $SHELL /bin/bash

Shell features • Two types of usage: • Command line - interactive • Shell script, usually non-interactive • Shell script defined as: • "Shell Script is series of commands written in plain text file. Shell script is just like batch file is MS-DOS but have more power than the MS-DOS batch file."

Shell features • Two types of commands: • Internal commands – built in the shell interpreter • External commands – calling other executable files • Almost everything applies to both command line usage and shell scripts

External commands • Execution of external programs – most common task External program: /bin/ls girtsf@linux tmp $ ls -l /lib total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables -rwxr-xr-x 1 root root 92716 Oct 14 13:10 ld-2.3.4.so -rwxr-xr-x 1 root root 22800 Oct 14 13:17 ld-linux.so.1 ...

External commands • Environment variable $PATH determines where to search for external programs. • girtsf@linux tmp $ echo $PATH /bin:/usr/bin:/usr/local/bin:/opt/bin • “:” as separator • Current directory “.” is usually not in PATH for security reasons.

External commands • girtsf@linux tmp $ echo $PATH /bin:/usr/bin:/usr/local/bin:/opt/bin • With /bin in path, typing “ls” suffices to run /bin/ls. • Example of unsetting path: • girtsf@linux tmp $ unset PATH girtsf@linux tmp $ ls bash: ls: No such file or directory girtsf@linux tmp $

Internal commands • A large list of built in commands, that are handled internally without running an external command • Most commonly used internal command is cd, used to change the current working directory: • girtsf@linux girtsf $ cd /tmp/ girtsf@linux tmp $

Aliasing • Aliasing is the process of assigning a command to a shorter “alias” • This allows you to type the shorter command instead of the longer one. • Aliasing is useful for changes that you want all of the time. • alias rm “rm –i” • Aliasing is similar to shell function definitions • dos2unix() { cat $1 | perl -pe 's/\r\n$/\n/g'; } • unix2dos() { cat $1 | perl -pe 's/\n$/\r\n/g'; }

Internal commands girtsf@linux tmp $ help GNU bash, version 2.05b.0(1)-release (i686-pc-linux-gnu) These shell commands are defined internally. Type `help' to see this list. Type `help name' to find out more about the function `name'. Use `info bash' to find out more about the shell in general. Use `man -k' or `info' to find out more about commands not in this list. A star (*) next to a name means that the command is disabled. %[DIGITS | WORD] [&] (( expression )) . filename : [ arg... ] [[ expression ]] alias [-p] [name[=value] ... ] bg [job_spec] bind [-lpvsPVS] [-m keymap] [-f fi break [n] builtin [shell-builtin [arg ...]] case WORD in [PATTERN [| PATTERN]. cd [-L|-P] [dir] command [-pVv] command [arg ...] ...

Helper utilities • Helper utilities – various small external programs that are helpful when working with shell scripts or command line • Called from shell (scripts or command line) • Somehow transforms input into output, based on the parameters

Helper utilities • cat - concatenate files and print on the standard output • Syntax: cat [file1] [file2] … [fileN] girtsf@linux etc $ cat gentoo-release shells Gentoo Base System version 1.4.16 # /etc/shells: valid login shells # $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/shells,v 1.5 2003/07/15 20:36:32 azarah Exp $ /bin/sh /bin/bash /bin/tcsh /bin/csh /bin/esh /bin/ksh /bin/zsh /bin/sash

Helper utilities • echo – displays a line of text • Besides a program /bin/echo, also usually built in the shell (takes precedence) • Syntax: echo [STRING] ... girtsf@linux girtsf $ echo quick brown fox quick brown fox Can be used to display environment variables girtsf@linux girtsf $ echo $HOME /home/girtsf

Helper utilities • wc - print the number of newlines, words, and bytes in files • wc [options] [file1] [file2] … [fileN] • By default, newlines, words and byte counts are displayed • Options • -c : print only byte count • -w : print only word count • -l : print only line count

Helper utilities • Example use of wc: girtsf@linux etc $ wc /etc/passwd 50 76 2257 /etc/passwd girtsf@linux etc $ wc -l /etc/passwd 50 /etc/passwd lines words bytes lines only

Helper utilities • grep - print lines matching a pattern • grep PATTERN [file1] [file2] … [fileN] • The lines that contain PATTERN are printed to standard output. • If no files are specified, input is taken from standard input (more later). • Advanced versions of grep allow using regular expressions in PATTERN.

Helper utilities • File “testfile” contains the following lines girtsf@linux girtsf $ cat testfile the quick brown fox jumped over the lazy dog • We search for “the”: girtsf@linux girtsf $ grep the testfile the quick brown the lazy dog • Only lines containing the substring “the” are printed.

Helper utilities • Some useful parameters for grep: • -i : ignore case (“the” finds “the”, “The”, “THE”,…) • -l : output only filenames that match, not the contents • -B <n> : output also n lines before the matching line • -A <n>: output also n lines after the matching line • See the man page (“man grep”) for all parameters

Helper utilities • Any program can be used as a helper program • More examples later

Connecting utilities with shell scripting • Standard I/O • I/O redirection to/from file • I/O redirection using a pipe • Backticks

Standard I/O • Every process, when run, has 3 already open data streams (file descriptors): • Standard input • Standard output • Standard error

Standard I/O • When run interactively (from command line), these streams are attached to the terminal they are running from • Standard input is attached to user’s keyboard input • Standard output is attached to user’s terminal output • Standard error, similarly to output, is attached to user’s terminal output • Usually referred to as stdin, stdout, stderr.

Standard output & error • “ls” command does not use input, but uses stdout, stderr. • The second line is the stdout from “ls” command: girtsf@linux etc $ ls -l /etc/passwd -rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd • The second line is from stderr from “ls” command: girtsf@linux etc $ ls -l /etc/asdfasdf ls: /etc/asdfasdf: No such file or directory • Both stdout and stderr simultaneously: girtsf@linux tmp $ ls -l /etc/passwd /etc/asdfasdf ls: /etc/asdfasdf: No such file or directory -rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd

Standard input • tee - read from standard input and write to standard output and files • Syntax: tee [File1] [File2] .. [FileN] • Example of tee taking user’s input from terminal and writing to 3 files: girtsf@linux tmp $ tee a b c some string^D some string girtsf@linux tmp $ cat a some string girtsf@linux tmp $ cat b some string girtsf@linux tmp $ cat c some string Inred – my input, ending with Control-D, which is the EOF (End of File) character. This input is read as standard input by tee.

I/O Redirection to/from file • By default, the 3 streams are attached to terminal • This can be overridden when executing the command and is called “redirection” • “>” specifies that stdout is redirected to file • “<“ specifies that stdin is taken from file • “2>” specifies that stderr is redirected to file

I/O Redirection to/from file • Syntax: • <cmd> [ > <file1>] [ < <file2> ] [ 2> <file3> ] • For those redirections that are specified, the respective stream will be attached to the specified file • None, one, two or all three types can be specified • If output file exists: > - replace file; >> - append to file

I/O Redirection to file • Example of stdout redirection to file girtsf@linux tmp $ ls -l /lib/ > direktorijas_saraksts girtsf@linux tmp $ cat direktorijas_saraksts total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables ...

I/O Redirection to file • Example of stdout redirection to file girtsf@linux tmp $ ls -l /asdf > direktorijas_saraksts ls: /asdf: No such file or directory girtsf@linux tmp $ cat direktorijas_saraksts • The file is empty, as no output was sent to stdout, as error message was send to stderr, which still was attached to user’s terminal

I/O Redirection to file • Example of stderr redirection to file girtsf@linux tmp $ ls -l /asdfasdf 2> errlog girtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directory • Now stderr was redirected to file and file contained the error message.

I/O Redirection to file • Example of stdout, stderr redirection to file girtsf@linux tmp $ ls -l /asdfasdf /lib 2>errlog >sar girtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directory girtsf@linux tmp $ cat sar /lib: total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables ...

I/O Redirection from file • Example of stdin redirection • First, we create file “a” with the following content the quick brown fox jumped over a quick brown fox • Use wc (word count) by not supplying the file name, but redirecting standard input girtsf@linux tmp $ wc < a 2 10 50

I/O Redirection with pipes • Task: given a file, output the total number of words in those lines, that contain substring “the”. • Example input: girtsf@linux girtsf $ cat testfile the quick brown fox jumped over the lazy dog • Lines 1 and 3 match, total number of words = 6.

I/O Redirection with pipes • Solution with redirection to files: • First find the lines, save them into temp file • Then use wc (word count) utility to count the number of words girtsf@linux girtsf $ grep the testfile > tmpfile girtsf@linux girtsf $ wc –w < tmpfile 6

I/O Redirection with pipes • Temporary file was used to redirect the standard output of grep to file • The standard input to wc was taken from temporary file • Easier way – connect the standard output of one program to standard input of another one directly

I/O Redirection with pipes • Syntax: program1 | program2 (| - pipe symbol) • Called “piping output from one program to another” • This example: girtsf@linux girtsf $ grep the testfile | wc -w 6 • No temporary files. Elegant!

Backticks • Backticks – reverse apostrophes “`” (usually the same key as tilde ~ character) • Using backticks sub-commands are executed, their result written in the place they are defined • Example: girtsf@linux tmp $ cd `echo /home` girtsf@linux home $ Substituted with “/home”

Helper utilities • We will examine the following utilities: • cut • sort • uniq • awk • sed

cut • cut - remove sections from each line of files • Syntax: cut [OPTION]... [FILE]... • Options: • -d DELIM : use DELIM instead of TAB character • -f LIST : output only these fields (delimited by DELIM) • -c LIST : output only these characters • See man page for more options

cut • Example – output second word on each line: • Delimiter: space “ “ • Fields: 2 girtsf@linux tmp $ cat a the quick brown fox jumped over a quick brown fox girtsf@linux tmp $ cut -f 2 -d ' ' a quick over

cut • Example – output characters 1-3, 5, 7-end • Use –c to choose the needed characters girtsf@linux tmp $ cat a thequick brown fox jumped over a quick brown fox girtsf@linux tmp $ cut -c 1-3,5,7- a theqick brown fox jume over a quick brown fox

sort • sort - sort lines of text files • sort [OPTION]... [FILE]... • Writes sorted concatenation of all FILE(s) to standard output. • Interesting options: • -r : reverse • -n : compare according to string numerical value • See man page for more options

sort • sort - sort text file reversed girtsf@linux tmp $ cat a fish dog animal bird girtsf@linux tmp $ sort -r a fish dog bird animal

sort • Sort numeric file (as text) girtsf@linux tmp $ cat a 5412 this line should go last 998 this line should go second 50 this line should go first 999 this line should go third girtsf@linux tmp $ sort a 50 this line should go first 5412 this line should go last 998 this line should go second 999 this line should go third

sort • Sort numeric file as numbers girtsf@linux tmp $ cat a 5412 this line should go last 998 this line should go second 50 this line should go first 999 this line should go third girtsf@linux tmp $ sort -n a 50 this line should go first 998 this line should go second 999 this line should go third 5412 this line should go last

uniq • uniq - remove duplicate lines from a sorted file • uniq [OPTION]... [INPUT [OUTPUT]] • Discards all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output). • Can be used together with sort, to get file without duplicate lines.

Just sorted: $ cat a | sort bird bird dog dog fish fish fly sort | uniq: $ cat a | sort | uniq bird dog fish fly uniq

awk • gawk (GNU awk) - pattern scanning and processing language. • Gawk is the GNU Project's implementation of the AWK programming language. It conforms to the definition of the language in the POSIX 1003.2 Command Language And Utilities Standard. This version in turn is based on the description in The AWK Programming Language, by Aho, Kernighan, and Weinberger, with the additional features found in the System V Release 4 version of UNIX awk.

awk • Complete language interpreter • Variables • User defined functions • … • Useful for small one-liners • Just some examples will be given • See man page or search for tutorials on the net

awk Task: sum all the numbers in the file: $ cat a 1 2 3 4 5 $ cat a | awk '{ sum += $1 } END { print sum }' 15 first field executed on every line executed at the end

Shell Programming