Automating System Administration Tasks

1. Automating System Administration Tasks Chapter 23

2. Understand what tasks lend themselves to automation. • Understand the tools available to automate system administration tasks. • Understand differences between shells, and shell programming languages. Chapter Goals

3. As we have seen, there are several repetitive tasks in system administration: • Downloading and installing patches • Installing accounts • Managing configuration files • Managing printers • Scanning log files • Monitoring security • Cleaning out temporary files. • These tasks lend themselves to automation using a variety of tools. What to automate?

4. Patches • RedHat up2date • Notre Dame’s “nd.installpatch” • Installing accounts • Acmaint • grep user_name /afs/nd.edu/common/etc/passwd >>/etc/passwd ; pwconv • Managing Configuration Files • rdist • expect • rsync • package What to automate?

5. Managing printers • Write a script to add printers for you • Scanning log files and monitoring security • Swatch • Logwatch • Write a script to watch for things of interest to you. • Cleaning out temporary files • cron • find What to automate?

6. Distributing system information (files) is a problem. • Small site --> small problem. • Large site --> LARGE problem. • Consider: Site with 4 system architectures: • Sun (250 hosts) • HP (250 hosts) • IBM (250 hosts) • PC’s with Linux (250 hosts) • How would you distribute a new version of sendmail to each host? Information Distribution

7. The first problem to tackle is “what should we distribute”? • Try to determine which files may be distributed to multiple system architectures (usually text files). • Try to determine what binaries are located on the system disks (not on network file systems). • Try to determine which files are scripts that would work on multiple architectures. • Try to determine which files are common to all hosts of a particular architecture. Information Distribution

8. Some typical examples: • System Binaries • sendmail • login – common to a single OS/architecture • ftpd • tcpd • Common configuration files • /etc/motd • sendmail.cf • password file • shadow file – may not be directly usable Information Distribution

9. The next problem to tackle is which hosts you want to distribute files to: • All hosts of an architecture? (Sparc, HP) • All hosts which are clients of a particular server? • All hosts which are null clients of a particular mail server? • All hosts in engineering? • The more files/architectures you have to distribute information to, the more complex the setup will be for the distribution method (and the more likely you will mess something up). Information Distribution

10. There are two methods of distributing system information: • The push method - a server pushes the information to the clients. • Advantages: • It is usually easy to run “unscheduled” push operations. • You have some control over when the process runs. • Disadvantages: • More difficult to update a single client. • Tainted files will be distributed just like good files. • Setup: • Need to build/maintain copies of files to distribute. • Need to build/maintain the distribution control file. Information Distribution

11. There are two methods of distributing system information: • The pull method - a client pulls the information from a server. • Advantages: • It is easy to run “unscheduled” updates. • Systems can be reset to a known state upon reboot. • Disadvantages: • More difficult to update clients all at once. • Administrator has no idea which hosts have updated. • Setup: • Need to build/maintain copies of files to distribute. • Need to build/maintain the distribution control file. Information Distribution

12. The push method • The rdist program is one example of a “push method” file distribution system. • Need to build a distfile • tells what systems to distribute the files to • tells which files to distribute to the systems • allows special operations to be performed • allows notification of success/failures • allows exceptions to the rules • may be run from a cron entry • Has a handy “no action” version that allows rules testing. Information Distribution

13. Rdist • Need to create a repository of files to distribute. • Good idea to use RCS/SCCS/CVS to keep track of versions, and changes. • Good idea to keep the repository “read only” so that bad versions of files do not get distributed. • Special caveat: • Rdist distributes the files with the ownership of the user who ran the rdist command. • This can be a major problem if not handled by the settings in the distfile! Information Distribution

14. # # distribute something to lab hosts # BASE=“/afs/nd.edu/user33/curt” SOL_CLIENTS=( snowwhite boole branin eckert noyce turing ) # # Files to distribute # lab6: $BASE/courses/cse444/lab6.init -> ${SOL_CLIENTS} install /etc/init.d/lab6; special "chmod 755 /etc/init.d/lab6"; Sample rdist file

15. # # distribute Solaris sendmail to grumpy, mail.cse), mail clients # distribute HPUX sendmail to geo # distribute AIX4 sendmail to yoyo and euclid BASE=/afs/nd.edu/wsadmin/cse_254 SRC=/afs/nd.edu/user14/csesoft/src SOL_CLIENTS=( aliens.dcrl atanasoff athena backus bashful berry boole branin) STANDALONE=( grumpy ) AIX_CLIENTS=(euclid yoyo) HPUX_CLIENTS=( geo ) SUNS=(${SOL_CLIENTS} ${STANDALONE} ${SERVER}) SERVER=( wizard ) ALL=(${SERVER} ${STANDALONE} ${SOL_CLIENTS} ${AIX_CLIENTS} ) Rdist Example

16. # # Files to distribute # sendmail.cf.solcli: $BASE/common/etc/mail/sendmail.cf -> ${SOL_CLIENTS} special "cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.old"; install /etc/mail/sendmail.cf.new; special "cp /etc/mail/sendmail.cf.new /etc/mail/sendmail.cf"; special "ln -s /etc/mail/sendmail.cf /etc/sendmail.cf"; special "rm /etc/mail/sendmail.cf.new"; sendmail.cf.server: $BASE/server/etc/mail/sendmail.cf -> ${SERVER} special "cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.old"; install /etc/mail/sendmail.cf.new; special "cp /etc/mail/sendmail.cf.new /etc/mail/sendmail.cf"; special "ln -s /etc/mail/sendmail.cf /etc/sendmail.cf"; special "rm /etc/mail/sendmail.cf.new"; Rdist example

17. sendmail.cf.stand: $BASE/grumpy/etc/mail/sendmail.cf -> ${STANDALONE} special "cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.old"; install /etc/mail/sendmail.cf.new; special "cp /etc/mail/sendmail.cf.new /etc/mail/sendmail.cf"; special "ln -s /etc/mail/sendmail.cf /etc/sendmail.cf"; special "rm /etc/mail/sendmail.cf.new"; sendmail.sun: $BASE/common/usr/lib/sendmail -> ${SUNS} special "cp /usr/lib/sendmail /usr/lib/sendmail.old"; install /usr/lib/sendmail.new; special "chgrp sys /usr/lib/sendmail"; special "chmod 6755 /usr/lib/sendmail"; special "/usr/local/bin/killj sendmail"; special "cp /usr/lib/sendmail.new /usr/lib/sendmail"; special "/usr/lib/sendmail -bd -q1h"; special "rm /usr/lib/sendmail.new"; Rdist example

18. sendmail.cf.aix: $SRC/sendmail/cf/cf/nd-csedept-AIX4-hidden.cf -> ${AIX_CLIENTS} special "cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.old"; install /etc/mail/sendmail.cf; special "ln -s /etc/mail/sendmail.cf /etc/sendmail.cf"; sendmail.aix: $SRC/sendmail/src/obj.AIX.4.000044673500/sendmail -> ${AIX_CLIENTS} special "cp /usr/lib/sendmail /usr/lib/sendmail.old"; install /usr/lib/sendmail; special "/usr/local/bin/killj sendmail"; special "/usr/lib/sendmail -bd -q1h"; sendmail.cf.hpux: $SRC/sendmail/cf/cf/nd-csedept-HPUX10-hidden.cf -> ${HPUX_CLIENTS} special "cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.old"; install /etc/mail/sendmail.cf; special "ln -s /etc/mail/sendmail.cf /etc/sendmail.cf"; Rdist example

19. sendmail.hpux: $SRC/sendmail/src/obj.HP-UX.10.2/sendmail -> ${HPUX_CLIENTS} special "cp /usr/lib/sendmail /usr/lib/sendmail.old"; install /usr/lib/sendmail; special "/usr/local/bin/killj sendmail"; special "/usr/lib/sendmail -bd -q1h"; Rdist example

20. The pull method • There are two common “pull” packages: • expect • expect uses ftp to copy files to the client • expect provides some directives to manage the ftp session. • can be run from a bootup script • can be run from a cron entry Information Distribution

21. #!/usr/local/src/gnu/bin/expect spawn /bin/ftp expect "ftp> " {send "open dopey.cselab.nd.edu\r"} expect "): " {send "gorp\r"} expect "Password:" {send "c@Ntheh@\r"} expect "ftp> " {send "bin\r"} expect "ftp> " {send "get np.tar /dev/null\r"} expect "ftp> " {send "quit\r"} exit Sample expect script

22. Another pull technology is the “package” command used with AFS. • package requires the administrator to set up a control file which contains a list of files which will be distributed. • package allows a noaction mode to test rules. • package could be run from cron (dangerous?). • package is usually run at system reboot. Information Distribution

23. # # CSE supplemental package file # Need these directories defined so that we can get to other entries # D / root root 755 D /afs root root 2755 D /etc root staff 755 D /etc/mail sys mail 755 D /etc/init.d root sys 775 D /etc/inet root sys 755 D /etc/rc3.d root sys 775 D /etc/rc2.d root sys 775 DR /var/tmp bin sys 1777 DR /tmp bin sys 1777 D /usr root sys 775 D /usr/bin root bin 775 D /usr/sbin root bin 775 D /usr/include root sys 775 D /usr/vice root other 755 D /usr/vice/etc root other 755 D /var root sys 755 D /var/spool root bin 755 D /var/spool/cron root sys 755 D /var/spool/cron/crontabs root sys 755 Sample package file

24. # # Some necessary links # Eases installation of new machines # LA /usr/include/X11 /afs/nd.edu/sun4x_55/usr/local/src/X11R6/usr/include/X11 LA /usr/lib/X11 /afs/nd.edu/sun4x_55/usr/local/src/X11R6/usr/lib/X11 # # Particular to this cluster # Package file, MOTD, and AFS cache size # F /etc/motd /afs/nd.edu/wsadmin/cse_254/211 root sys 644 # # For all CSE clients (i.e. non-wizard) # F /.rhosts /afs/nd.edu/wsadmin/cse_254/common root sys 444 F /.cshrc /afs/nd.edu/wsadmin/cse_254/common root sys 444 F /.profile /afs/nd.edu/wsadmin/cse_254/common root sys 444 F /var/spool/cron/crontabs/root /afs/nd.edu/wsadmin/cse_254/common root other 400 Sample package file

25. # Replace some /etc F /etc/passwd /afs/nd.edu/wsadmin/cse_254/211server root other 644 F /etc/resolv.conf /afs/nd.edu/wsadmin/cse_254/211server root other 644 F /etc/pam.conf /afs/nd.edu/wsadmin/cse_254/common.26 root other 644 FQ /etc/inet/hosts /afs/nd.edu/wsadmin/cse_254/211server root other 444 LA /etc/hosts /etc/inet/hosts root other 644 F /etc/inet/services /afs/nd.edu/wsadmin/cse_254/common root other 644 LA /etc/services /etc/inet/services root other 644 F /etc/syslog.conf /afs/nd.edu/wsadmin/cse_254/common root other 644 F /etc/init.d/cse_client /afs/nd.edu/wsadmin/cse_254/common root other 755 LA /etc/rc3.d/S99cse_client /etc/init.d/cse_client root other 755 F /usr/etc/ifstatus /afs/nd.edu/wsadmin/cse_254/common.26 root other 755 F /etc/inet/inetd.conf /afs/nd.edu/wsadmin/cse_254/common root sys 444 LA /etc/inetd.conf /etc/inet/inetd.conf root other 444 F /usr/sbin/in.ftpd /afs/nd.edu/wsadmin/cse_254/common.26 bin bin 555 F /etc/ftpaccess /afs/nd.edu/wsadmin/cse_254/common.26 bin bin 555 F /usr/bin/top /afs/nd.edu/wsadmin/cse_254/common bin bin 4755 ## FAQ /usr/lib/sendmail /afs/nd.edu/wsadmin/cse_254/common/usr/lib/sendmail root other 4555 F /usr/lib/sendmail.hf /afs/nd.edu/wsadmin/cse_254/common root other 555 F /etc/mail/sendmail.cf /afs/nd.edu/wsadmin/cse_254/211 root other 444 LA /etc/sendmail.cf /etc/mail/sendmail.cf root other 444 Sample package file

26. Other distribution methods • NIS - centralized database, not really a way to “distribute” files. The files are available to other hosts, but they are located on a (few) server(s). • MIT’s Hesoid - never caught on, difficult to build configuration files. Uses DNS to provide information from a central server. • Netinfo - NeXT computer’s version of NIS. • Rsync • Cfengine Information Distribution

27. Many automation tasks are handled using shell (or other) scripts. • Shell scripts are generally portable across multiple versions of operating systems. • Other scripting languages, such as Python, or Perl are much more powerful, but not always installed on systems by default. Shells

28. Shells • UNIX provides several shell programs for the users. • These programs are the user’s interface with the system software. • The shell programs read commands from the user terminal and interpret or execute them as required. • Some of the more popular shells available are: • sh - the original Bourne shell • csh - the Berkeley C shell • bash - the Bourne Again SHell (an updated sh) • ksh - the Korn shell • tcsh - an updated version of the C shell. Shell Programming

29. Shells • Shell selection is a highly subjective user decision. • While the shells are primarily a command interpreter, they also provide an interpreted programming language which users can employ to make their lives easier. • Different shells implement different programming languages. • Unfortunately, system administrators have to be proficient with several shell languages in order to manage the system. • Over the next few lectures we will look at the “sh” and “csh” shells. In particular we will look at using these shells to write shell scripts which will assist us with our system administration tasks. Shell Programming

30. Shells • Most of you use a variant of the Cshell (csh or tcsh). • The Cshell provides some job control, and command line editing options not available under the Bourne shell (sh). • Tcsh provides more advanced command line editing and parsing than csh...but tcsh is not a standard shell on all versions of Unix. • Under Solaris the root account uses the bourne shell located in /sbin/sh. • The bourne shell is not my favorite shell, but the examples I give will be sh based. • If you are serious about system administration, I suggest you become familiar with both sh and csh environments. Shell Programming

31. Common Shell Features • All shells have some common features: • Shells allow interactive or background processing. • Shells allow input/output redirection. • Shells allow “pipes” between commands. • Shells allow wildcard matching. • Shells assign special meaning to some characters. • Shells use variables to store values. • The common invocation of a shell command is: • command[options] arguments Shell Programming

32. Example Script Invocations nispopulate -u -F -p /nisstuff/update -d cselab.nd.edu. /etc/init.d/lab10 start which talk cd /tmp basename /afs/nd.edu/user33/curt Shell Programming

33. Built-in Shell Operations • Shells all have some built-in commands which we can use when writing shell scripts: • echo - write a message on standard output (CSH). • Been hit by root kit? Use echo! • read - read something from standard input • test - test various conditions • expr - evaluate an expression • true - return a true value • false - return a false value • wait - wait for something (good) to happen • sleep - stop for some period of time Shell Programming

34. Shell Variables (Built-in) • All shells have some built-in variables (reserved names) which can be useful when we write shell scripts: • PATH - The system search path • LOGNAME - The users login name • TERM - The kind of terminal the user is at • HOME - The users home directory • MAIL - The location of the users mailbox • PS1 - The primary prompt string for the user • LD_LIBRARY_PATH - The path to system libraries. Shell Programming

35. Shell Variables (user assigned) • To assign a values to a variables: • variable=value (set variable=value for CSH) • for example fruit=apple, or i=1, set name=$USER • To access the value of the variable, prepend a “$” : echo fruit fruit echo $fruit apple echo “User: $USER” User: curt Shell Programming

36. Shell Variables • You can also set variables by executing commands: menu=`cat food` now=`date` • It is also possible to assign string values to variables: PS1=“`uname -n` - Yes Master: “ echo $PS1 grumpy - Yes Master: • Must place multi-word strings in double quotes Shell Programming

37. Metacharacters • All shells treat certain characters as special. • $ ^ ; & { } [ ] ‘ ` * ( ) | < > newline space tab • To use these characters in scripts they must be quoted or escaped. • escape a character by placing a backslash in front of it. • single quotes protect text from any substitutions. • back quotes cause the shell to perform command substitution • double quotes protect against wildcard substitution Shell Programming

38. Escaping Characters: $ ( ) syntax error: `)' unexpected $ \(\) (): not found Single Quotes $ it's $cmd > ' its $cmd^J: not found $ it\'s $cmd it's: not found Shell Programming

39. Back Quotes $ HN=uname -n -n: not found $ HN=`uname -n` $ echo $HN grumpy $ who am i curt pts/5 Nov 11 13:47 $ 'who am i' who am i: not found $ `who am i` curt: not found Shell Programming

40. Double Quotes $ grep Things to do todolist grep: can't open to grep: can't open do todolist:Things to do: $ grep "Things to do" todolist Things to do: Shell Programming

41. Shell test features • All shells provide test mechanisms: • This feature allows us to do conditional processing within shell scripts. • In sh we can test for: • numeric values: if [ $# -eq 0 ] ; then if ( $# = 0 ) then (CSH version) general form: N <primitive> M -eq, -ne, -gt, -lt, -ge, -le = != > < >= <= -o (or) -a (and) Shell Programming

42. Test Conditions • The shells also provides a way to test the external environment: -r file (read permission) -w file (write permission) -x file (execute permission) -e file (file exists) -o file (user owns file) -z file (file is empty) -f file (plain file) -d file (file is a directory) if [ -e /usr/lab/tmp/”$AFSID”.start ] ; then if [ ! -d /usr/lab/tmp/”$AFSID” ] ; then Shell Programming

43. Shell test features • string equality if [ $LOGNAME = “root” ] ; then general form: S <primitive> R ; or <primitive> S primitives: =, != -z - is string zero length -n - is string non zero length • Testing strings can produce “odd” results: • number=1 • nombre=‘ 1’ test $number = $nombre evaluates true (spaces eaten) test “$number” = “$nombre” evaluates false Shell Programming

44. Shell Math • The shell also allows mathematical operations on variables: • The expr command evaluates it’s arguments. • General form: expr variable operation variable • operations: +, -, *, /, % (remainder) • sum=`expr $1 + $2` Shell Programming

45. Conditional Processing • The shells all provide a way to do conditional processing: if [ condition ] ; then if ( condition) then statements else (or else if) statements fi endif Shell Programming

46. Conditional Processing for variable in list # BOURNE SHELL do statements done foreach i( list ) # CSH statements end Shell Programming

47. Conditional Processing while condition #BOURNE while condition # CSH do statements statements done end until condition do statements done Shell Programming

48. Conditional Processing SH case variable in pattern 1) statements ;; pattern 2) statements ;; *) #default case statements ;; esac Shell Programming

49. Conditional Processing CSH switch ( variable ) case pattern 1: statements breaksw pattern 2: statements breaksw default : statements breaksw endsw Shell Programming

50. Argument Handling • The shell defines a method of handling arguments: • $$ is the process id of the shell program • The name of the shell program itself is $0 • Argument variables are numbered in the order passed: • $1 $2 $3 $4 ... • $* and $@ access all variables • $* lumps all variables into a single string • $@ keeps variables as separate entities • $# is set to the argument count Shell Programming