Scripting on Linux

1. Origins of scripting languages Learning scripting languages A simple example of a shell script A script with a loop and debugging Processing a table of data by selecting rows and columns Doing arithmetic internally within the shell Perl, Python, Ruby etc. Scheduling scripted jobs using cron Scripting on Linux

2. Hardwired programs and embedded OS allowed little system control. Job control language JCL allowed IBM mainframe operators to "batch" a sequence of programs into a routine job, e.g. deducting payments made from accounts outstanding. Benefits: * greater accuracy of job control * less typing * ability to restart job from initial data when system goes down part way through. Mainframe features reinvented on smaller/cheaper machines. Origins of scripting languages

3. TSO/Clist (IBM on-line), PCL (Prime Primos), DCL (DEC VAX/VMS), Aegis (Apollo DomainOS). MS-DOS Batch File (.BAT) etc. Microsoft froze .BAT language early 1990s. Early versions of MS Basic packaged with MS-DOS, later versions sold separately. Unix opened development and transfer of ideas between shells. * Bourne Shell: /bin/sh * 'C' Shell: /bin/csh * Korn Shell: /bin/ksh * Ash Shell: /bin/ash * GNU Bash Shell: /bin/sh and /bin/bash Unix shell scripts have usual 3rdG features, including loop constructs, variables, arrays, branches and functions. Command languages developed

4. Many features shared between shell languages and 'C', Perl, Python etc. The 80/20 rule: 80% of the benefit derives from knowledge of the 20% most useful subset. Programmers learn by: * Reading tutorials, books, reference material * Studying code examples * Running modified code examples * Many small experiments * Knowing how to program using a similar language type Learning scripting languages

5. Those requiring an in-depth understanding of these languages will need to read the books and on-line tutorials and carry out a comprehensive series of programming exercises. In other cases a usable subset of knowledge can be obtained by reading the source code of existing programs and executing these, and by conducting small experiments supplemented with tactical use of the reference information provided with languages. Programming knowledge can grow on an as-needed basis. Learning scripting languages 2

6. A simple shell script example

7. The Unix shell convention is for a program which exits successfully to end with a return of 0 (which the shell considers true), and for an error exit to result in a return of 1 or more (which the shell considers false). This is the opposite way round to how this is done within 'C'. The number N returned by a program (e.g. using the bash exit N statement) is different from the standard output of the same program. The shell command: echo $? outputs the return code of the last foreground command. Return values and tests

8. A script with a loop and debugging

9. menu script part 2

10. Here are some records from the may.logins input file: reboot ~ Tue May 11 13:14 shutdown ~ Tue May 11 13:15 usr11361 console Tue May 11 09:04 - 10:22 (01:18)‏ usr11187 console Mon May 10 18:53 - 20:30 (01:36)‏ usr11187 console Mon May 10 18:50 - 18:53 (00:02)‏ usr11187 console Mon May 10 18:38 - 18:50 (00:12)‏ usr11513 console Mon May 10 15:15 - 16:27 (01:11)‏ usr11451 console Mon May 10 12:11 still logged in usr11456 console Mon May 10 10:53 - 15:14 (04:21)‏ usr11138 console Mon May 10 09:03 - 10:40 (01:36)‏ usr12069 console Sat May 8 11:05 - 09:01 (1+21:55)‏ usr12069 console Sat May 8 11:00 - 11:04 (00:04)‏ This gives us time logged, one record per login session. A table with rows and columns

11. This combines a number of features of previous examples, using awk, grep and sed filters to access specific rows then columns, and to exclude unwanted data from the analysis. The input data is a set of login records. This application was used to analyse average usage of 20 workstations during particular months. Login analysis script 2

12. Login analysis script 3

13. Smaller or earlier Unix shells don't have builtin arithmetic operators, so they farm this job out to external programs such as expr as we saw above. This can be done using the Bash shell let builtin, as in the following example script: Doing arithmetic internally within the shell

14. Shell scripts provide glue logic together with other utilities, pipelines and redirection. Advantage: Very rapid development of systems/network administration and automated operations. Disadvantages: Slow, not for very large programs, non-portable. Having to load and execute external programs very many times ? Adding 200 user login accounts to a system: * Shell: 1 hour to run (1992), helped us learn how to do it * Perl: ('93) 2 minutes to run, we already knew how to do it * 'C' ('94) 6 seconds but 10 times as long to write the program. Advanced scripting: Perl, Python, Ruby etc.

15. Fully portable languages: Perl, Python and Ruby used to handle simple scripting-type applications or more complex requirements, e.g. object-oriented webapp development. These languages build upon features in shell languages, using similar syntax for many purposes, e.g. handling regular expressions. These languages trade machine for programmer efficiency, compare against 'C', 'C++' . Can combine benefits by using 'C' modules in scripting interpreter or split application design after profiling to find where cycles used. Advanced scripting: Perl, Python, Ruby 2

16. The login analysis program described above was rewritten in Perl. Advanced scripting: Perl, Python, Ruby 3

17. This program runs much faster than the shell script, because everything is done inside the same process. There are many syntactic similarities, but Perl borrows array and loop notation from 'C' and some other notation from grep, awk and sed. As in Bash, $ is used to introduce single (scalar) variables and @ is used for arrays. $_ is used for a default scalar variable, so that the substitution operation e.g: s/\)//; which strips a closing round bracket from a string, doesn't state the string it modifies. Advanced scripting: Perl, Python, Ruby 4

18. If scripts are the key to systems automation, how do we cause these to be run at set times of the day, or on set days of the week or month ? The program we use for this is called cron. Cron is a background process which runs other programs at set times. It is controlled using a file called crontab. Crontab contains entries to run all scripts placed in directories: /etc/cron.hourly /etc/cron.daily, /etc/cron.weekly etc. Scheduling cron jobs on Linux 1

19. # /etc/crontab: system-wide crontab # Unlike any other crontab you don't have to run the `crontab' # command to install the new version when you edit this file # and files in /etc/cron.d. These files also have username fields, # that none of the other crontabs do. SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )‏ 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )‏ 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )‏ # local modifications # update spam black and whitelist DNS zones every 10 minutes 01,11,21,31,41,51 * * * * root /usr/local/bin/update_dnsfiles # create and email report of weekly spam rejections 22 10 * * 6 root /usr/local/bin/mrejects Scheduling cron jobs on Linux 2