1 / 94

Shell Programming

Shell Programming. Asoc. Prof. Guntis Barzdins Asist. Girts Folkmanis. Lecture outline. Shell features Helper utilities, introduction Connecting utilities with shell scripting Helper utilities in detail Piping, advanced examples Shell scripts as files, Internal shell commands.

Download Presentation

Shell Programming

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Shell Programming Asoc.Prof. Guntis Barzdins Asist. Girts Folkmanis

  2. Lecture outline • Shell features • Helper utilities, introduction • Connecting utilities with shell scripting • Helper utilities in detail • Piping, advanced examples • Shell scripts as files, Internal shell commands

  3. Shell features • We will talk about bash, there might be differences for other shells. • bash - GNU Bourne-Again Shell • Authors: Brian Fox and Chet Ramey • Free Software Foundation • Popular in different distributions • Tip: To find your current shell, type following command $ echo $SHELL /bin/bash

  4. Shell features • Two types of usage: • Command line - interactive • Shell script, usually non-interactive • Shell script defined as: • "Shell Script is series of commands written in plain text file. Shell script is just like batch file is MS-DOS but have more power than the MS-DOS batch file."

  5. Shell features • Two types of commands: • Internal commands – built in the shell interpreter • External commands – calling other executable files • Almost everything applies to both command line usage and shell scripts

  6. External commands • Execution of external programs – most common task External program: /bin/ls girtsf@linux tmp $ ls -l /lib total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables -rwxr-xr-x 1 root root 92716 Oct 14 13:10 ld-2.3.4.so -rwxr-xr-x 1 root root 22800 Oct 14 13:17 ld-linux.so.1 ...

  7. External commands • Environment variable $PATH determines where to search for external programs. • girtsf@linux tmp $ echo $PATH /bin:/usr/bin:/usr/local/bin:/opt/bin • “:” as separator • Current directory “.” is usually not in PATH for security reasons.

  8. External commands • girtsf@linux tmp $ echo $PATH /bin:/usr/bin:/usr/local/bin:/opt/bin • With /bin in path, typing “ls” suffices to run /bin/ls. • Example of unsetting path: • girtsf@linux tmp $ unset PATH girtsf@linux tmp $ ls bash: ls: No such file or directory girtsf@linux tmp $

  9. Internal commands • A large list of built in commands, that are handled internally without running an external command • Most commonly used internal command is cd, used to change the current working directory: • girtsf@linux girtsf $ cd /tmp/ girtsf@linux tmp $

  10. Aliasing • Aliasing is the process of assigning a command to a shorter “alias” • This allows you to type the shorter command instead of the longer one. • Aliasing is useful for changes that you want all of the time. • alias rm “rm –i” • Aliasing is similar to shell function definitions • dos2unix() { cat $1 | perl -pe 's/\r\n$/\n/g'; } • unix2dos() { cat $1 | perl -pe 's/\n$/\r\n/g'; }

  11. Internal commands girtsf@linux tmp $ help GNU bash, version 2.05b.0(1)-release (i686-pc-linux-gnu) These shell commands are defined internally. Type `help' to see this list. Type `help name' to find out more about the function `name'. Use `info bash' to find out more about the shell in general. Use `man -k' or `info' to find out more about commands not in this list. A star (*) next to a name means that the command is disabled. %[DIGITS | WORD] [&] (( expression )) . filename : [ arg... ] [[ expression ]] alias [-p] [name[=value] ... ] bg [job_spec] bind [-lpvsPVS] [-m keymap] [-f fi break [n] builtin [shell-builtin [arg ...]] case WORD in [PATTERN [| PATTERN]. cd [-L|-P] [dir] command [-pVv] command [arg ...] ...

  12. Helper utilities • Helper utilities – various small external programs that are helpful when working with shell scripts or command line • Called from shell (scripts or command line) • Somehow transforms input into output, based on the parameters

  13. Helper utilities • cat - concatenate files and print on the standard output • Syntax: cat [file1] [file2] … [fileN] girtsf@linux etc $ cat gentoo-release shells Gentoo Base System version 1.4.16 # /etc/shells: valid login shells # $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/shells,v 1.5 2003/07/15 20:36:32 azarah Exp $ /bin/sh /bin/bash /bin/tcsh /bin/csh /bin/esh /bin/ksh /bin/zsh /bin/sash

  14. Helper utilities • echo – displays a line of text • Besides a program /bin/echo, also usually built in the shell (takes precedence) • Syntax: echo [STRING] ... girtsf@linux girtsf $ echo quick brown fox quick brown fox Can be used to display environment variables girtsf@linux girtsf $ echo $HOME /home/girtsf

  15. Helper utilities • wc - print the number of newlines, words, and bytes in files • wc [options] [file1] [file2] … [fileN] • By default, newlines, words and byte counts are displayed • Options • -c : print only byte count • -w : print only word count • -l : print only line count

  16. Helper utilities • Example use of wc: girtsf@linux etc $ wc /etc/passwd 50 76 2257 /etc/passwd girtsf@linux etc $ wc -l /etc/passwd 50 /etc/passwd lines words bytes lines only

  17. Helper utilities • grep - print lines matching a pattern • grep PATTERN [file1] [file2] … [fileN] • The lines that contain PATTERN are printed to standard output. • If no files are specified, input is taken from standard input (more later). • Advanced versions of grep allow using regular expressions in PATTERN.

  18. Helper utilities • File “testfile” contains the following lines girtsf@linux girtsf $ cat testfile the quick brown fox jumped over the lazy dog • We search for “the”: girtsf@linux girtsf $ grep the testfile the quick brown the lazy dog • Only lines containing the substring “the” are printed.

  19. Helper utilities • Some useful parameters for grep: • -i : ignore case (“the” finds “the”, “The”, “THE”,…) • -l : output only filenames that match, not the contents • -B <n> : output also n lines before the matching line • -A <n>: output also n lines after the matching line • See the man page (“man grep”) for all parameters

  20. Helper utilities • Any program can be used as a helper program • More examples later

  21. Connecting utilities with shell scripting • Standard I/O • I/O redirection to/from file • I/O redirection using a pipe • Backticks

  22. Standard I/O • Every process, when run, has 3 already open data streams (file descriptors): • Standard input • Standard output • Standard error

  23. Standard I/O • When run interactively (from command line), these streams are attached to the terminal they are running from • Standard input is attached to user’s keyboard input • Standard output is attached to user’s terminal output • Standard error, similarly to output, is attached to user’s terminal output • Usually referred to as stdin, stdout, stderr.

  24. Standard output & error • “ls” command does not use input, but uses stdout, stderr. • The second line is the stdout from “ls” command: girtsf@linux etc $ ls -l /etc/passwd -rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd • The second line is from stderr from “ls” command: girtsf@linux etc $ ls -l /etc/asdfasdf ls: /etc/asdfasdf: No such file or directory • Both stdout and stderr simultaneously: girtsf@linux tmp $ ls -l /etc/passwd /etc/asdfasdf ls: /etc/asdfasdf: No such file or directory -rw-r--r-- 1 root root 2257 Oct 22 13:35 /etc/passwd

  25. Standard input • tee - read from standard input and write to standard output and files • Syntax: tee [File1] [File2] .. [FileN] • Example of tee taking user’s input from terminal and writing to 3 files: girtsf@linux tmp $ tee a b c some string^D some string girtsf@linux tmp $ cat a some string girtsf@linux tmp $ cat b some string girtsf@linux tmp $ cat c some string Inred – my input, ending with Control-D, which is the EOF (End of File) character. This input is read as standard input by tee.

  26. I/O Redirection to/from file • By default, the 3 streams are attached to terminal • This can be overridden when executing the command and is called “redirection” • “>” specifies that stdout is redirected to file • “<“ specifies that stdin is taken from file • “2>” specifies that stderr is redirected to file

  27. I/O Redirection to/from file • Syntax: • <cmd> [ > <file1>] [ < <file2> ] [ 2> <file3> ] • For those redirections that are specified, the respective stream will be attached to the specified file • None, one, two or all three types can be specified • If output file exists: > - replace file; >> - append to file

  28. I/O Redirection to file • Example of stdout redirection to file girtsf@linux tmp $ ls -l /lib/ > direktorijas_saraksts girtsf@linux tmp $ cat direktorijas_saraksts total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables ...

  29. I/O Redirection to file • Example of stdout redirection to file girtsf@linux tmp $ ls -l /asdf > direktorijas_saraksts ls: /asdf: No such file or directory girtsf@linux tmp $ cat direktorijas_saraksts • The file is empty, as no output was sent to stdout, as error message was send to stderr, which still was attached to user’s terminal

  30. I/O Redirection to file • Example of stderr redirection to file girtsf@linux tmp $ ls -l /asdfasdf 2> errlog girtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directory • Now stderr was redirected to file and file contained the error message.

  31. I/O Redirection to file • Example of stdout, stderr redirection to file girtsf@linux tmp $ ls -l /asdfasdf /lib 2>errlog >sar girtsf@linux tmp $ cat errlog ls: /asdfasdf: No such file or directory girtsf@linux tmp $ cat sar /lib: total 4035 -rwxr-xr-x 1 root root 7488 Oct 6 12:33 cpp drwxr-xr-x 13 root root 1024 Oct 25 15:57 dev-state drwxr-xr-x 2 root root 1024 Jun 28 09:53 evms drwxr-xr-x 2 root root 2048 Aug 23 15:25 iptables ...

  32. I/O Redirection from file • Example of stdin redirection • First, we create file “a” with the following content the quick brown fox jumped over a quick brown fox • Use wc (word count) by not supplying the file name, but redirecting standard input girtsf@linux tmp $ wc < a 2 10 50

  33. I/O Redirection with pipes • Task: given a file, output the total number of words in those lines, that contain substring “the”. • Example input: girtsf@linux girtsf $ cat testfile the quick brown fox jumped over the lazy dog • Lines 1 and 3 match, total number of words = 6.

  34. I/O Redirection with pipes • Solution with redirection to files: • First find the lines, save them into temp file • Then use wc (word count) utility to count the number of words girtsf@linux girtsf $ grep the testfile > tmpfile girtsf@linux girtsf $ wc –w < tmpfile 6

  35. I/O Redirection with pipes • Temporary file was used to redirect the standard output of grep to file • The standard input to wc was taken from temporary file • Easier way – connect the standard output of one program to standard input of another one directly

  36. I/O Redirection with pipes • Syntax: program1 | program2 (| - pipe symbol) • Called “piping output from one program to another” • This example: girtsf@linux girtsf $ grep the testfile | wc -w 6 • No temporary files. Elegant!

  37. Backticks • Backticks – reverse apostrophes “`” (usually the same key as tilde ~ character) • Using backticks sub-commands are executed, their result written in the place they are defined • Example: girtsf@linux tmp $ cd `echo /home` girtsf@linux home $ Substituted with “/home”

  38. Helper utilities • We will examine the following utilities: • cut • sort • uniq • awk • sed

  39. cut • cut - remove sections from each line of files • Syntax: cut [OPTION]... [FILE]... • Options: • -d DELIM : use DELIM instead of TAB character • -f LIST : output only these fields (delimited by DELIM) • -c LIST : output only these characters • See man page for more options

  40. cut • Example – output second word on each line: • Delimiter: space “ “ • Fields: 2 girtsf@linux tmp $ cat a the quick brown fox jumped over a quick brown fox girtsf@linux tmp $ cut -f 2 -d ' ' a quick over

  41. cut • Example – output characters 1-3, 5, 7-end • Use –c to choose the needed characters girtsf@linux tmp $ cat a thequick brown fox jumped over a quick brown fox girtsf@linux tmp $ cut -c 1-3,5,7- a theqick brown fox jume over a quick brown fox

  42. sort • sort - sort lines of text files • sort [OPTION]... [FILE]... • Writes sorted concatenation of all FILE(s) to standard output. • Interesting options: • -r : reverse • -n : compare according to string numerical value • See man page for more options

  43. sort • sort - sort text file reversed girtsf@linux tmp $ cat a fish dog animal bird girtsf@linux tmp $ sort -r a fish dog bird animal

  44. sort • Sort numeric file (as text) girtsf@linux tmp $ cat a 5412 this line should go last 998 this line should go second 50 this line should go first 999 this line should go third girtsf@linux tmp $ sort a 50 this line should go first 5412 this line should go last 998 this line should go second 999 this line should go third

  45. sort • Sort numeric file as numbers girtsf@linux tmp $ cat a 5412 this line should go last 998 this line should go second 50 this line should go first 999 this line should go third girtsf@linux tmp $ sort -n a 50 this line should go first 998 this line should go second 999 this line should go third 5412 this line should go last

  46. uniq • uniq - remove duplicate lines from a sorted file • uniq [OPTION]... [INPUT [OUTPUT]] • Discards all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output). • Can be used together with sort, to get file without duplicate lines.

  47. Just sorted: $ cat a | sort bird bird dog dog fish fish fly sort | uniq: $ cat a | sort | uniq bird dog fish fly uniq

  48. awk • gawk (GNU awk) - pattern scanning and processing language. • Gawk is the GNU Project's implementation of the AWK programming language. It conforms to the definition of the language in the POSIX 1003.2 Command Language And Utilities Standard. This version in turn is based on the description in The AWK Programming Language, by Aho, Kernighan, and Weinberger, with the additional features found in the System V Release 4 version of UNIX awk.

  49. awk • Complete language interpreter • Variables • User defined functions • … • Useful for small one-liners • Just some examples will be given • See man page or search for tutorials on the net

  50. awk Task: sum all the numbers in the file: $ cat a 1 2 3 4 5 $ cat a | awk '{ sum += $1 } END { print sum }' 15 first field executed on every line executed at the end

More Related