Syslog and Log Files

1. Syslog and Log Files Chapter 11

2. Introduction • The accounting system, the kernel, and carious utilities emit data that is logged and eventually ends up on your finite-sized disks. • Most of the data has a limited useful lifetime and needs to be summarized, compressed, archived, and eventually thrown away. Chapter 11 - Syslog and Log Files

3. 1. Logging Policies • Logging policies vary from site to site. • Common schemes include: • Throw it all away immediately • Reset log files at periodic intervals • Rotate log files, keeping data for a fixed time. • Compress and archive logs to tape or other permanent media. • Whatever scheme you select, maintenance of log files should be automated with cron (Chapter 9) Chapter 11 - Syslog and Log Files

4. 1. Logging Policies • Throwing away log files • We do not recommend throwing away all logging information. • Log files provide important evidence of break ins and are helpful for alerting you to hardware and software problems. • In general, given enough disk space, data should be kept for at least a moth and then discarded. • If you need further back than that, you can go to backups. Chapter 11 - Syslog and Log Files

5. 1. Logging Policies • Rotating log files • Most sites store each day’s log information on disk • Sometimes in a compressed format. • Keeping them online allows them to be searched (with grep) • Dedicating a disk partition to logs is also an intelligent decision. Chapter 11 - Syslog and Log Files

6. 1. Logging Policies • One common way to limit disk usage is to rotate log files. • If a log file is called logfile, for example, the backup copies might be called logfile.1, logfile.2, and so on. • You could use a shell script to implement this rotation #!/bin/sh cd /var/log mv logfile.2 logfile.3 mv logfile.1 logfile.2 mv logfile logfile.1 cat /dev/null > logfile chmod 600 logfile • Ownership is important, so you may need a chown added. Chapter 11 - Syslog and Log Files

7. 1. Logging Policies • Many systems supply an off-the-shelf log rotation script that is run out of cron. • By all means, use the standard script if it does what you want. • If your system doesn’t supply a rotation system, we suggest that you use a Perl script called rotz written by Matt Segur and Michael Bernstein for this purpose. • It’s available from www.admin.com Chapter 11 - Syslog and Log Files

8. 1. Logging Policies • Archiving log files • Some sites must archive all accounting data and log files as a matter of policy, perhaps to provide data for a potential audit. • Log files should always be included in your regular backup sequence. • They may also be archived to a separate tape series. Chapter 11 - Syslog and Log Files

9. 2. Finding Log Files • UNIX is often criticized for being inconsistent, and indeed it is. • Just take a look at log files and you’re sure to find some with names like maillog, some like ftp.log, and maybe even some like lpNet, lpd-errs, or console-log. • In addition to having random names, some are often scattered across directories and filesystems. • This section attempts to help you find all the files that are quietly taking over your disk • And suggest a granularity for dealing with each. Chapter 11 - Syslog and Log Files

10. 2. Finding Log Files • To locate your log files, read your system’s startup scripts (/etc/rc*, /etc/rc.d/*, or /etc/init.d/*) to see if logging is turned on when daemons are run. • Table 11.1 compiles information about some of the more common log files on our example systems. • Filenames are relative to /var/adm or /var/log unless otherwise noted. • Log files usually have mode 644 • some sites reduce permissions to 640 or 600 • sulog, sudo.log and others should be 600 Chapter 11 - Syslog and Log Files

11. 2. Finding Log Files Chapter 11 - Syslog and Log Files

12. 3. Files NOT to Manage • You might be tempted to manage all log files with a rotation and archiving scheme. • But there are two files that you should not touch • /var/adm/lastlog • records each user’s last login. It is a sparse file indexed by UID (and it has holes) • /etc/utmp • utmp attempts to keep a record of each user that is currently logged in Chapter 11 - Syslog and Log Files

13. 4. Vendor Specifics • Vendors seem to have hidden log files all over the disk. • Careful detective work with your daemon’s config files and your syslog configuration file will find many of them. • This section details some of the more obscure nooks and crannies in which log files have been hidden. Chapter 11 - Syslog and Log Files

14. 4. Vendor Specifics • Solaris has the most disorganized collection of log files ever. • With a directory called /var/log it shouldn’t be so hard. • Here are a few places to look • /var/log/* • /var/cron/log • /var/lp/logs/* • /var/saf/_log • /var/saf/zsmon/log • /var/adm/{messages, aculog, sulog, vold.log, wtmpx} Chapter 11 - Syslog and Log Files

15. 4. Vendor Specifics • HP-UX log files are in /var/adm • There are a lot of odd little mystery files in this directory, many of which are not log files, • so be careful what you touch • By default, all log entries submitted via syslog go into the /var/adm/syslog directory. Chapter 11 - Syslog and Log Files

16. 4. Vendor Specifics • Red Hat gets a gold star for logging sanity. • Not only are logs clearly named and consistently stored in /var/log, • but Red Hat also provides a superior tool, logrotate, for rotating, truncating, and managing them • new software packages can drop a configuration file into /etc/logrotate.d directory to set up a management strategy for their log files. Chapter 11 - Syslog and Log Files

17. 4. Vendor Specifics • FreeBSD is another prize winner in the realm of logging. • Log files are generally found in /var/log, • although cron’s log is kept in /var/cron • and accounting files are kept in /var/account • The newsyslog utility is responsible for managing and rotating logs. • It runs out of cron and takes its marching orders from /etc/newsyslog.conf Chapter 11 - Syslog and Log Files

18. 5. syslog: The System Event Logger • syslog, originally written by Eric Allman, is a comprehensive logging system. • Many vendors use syslog to manage the information generated by the kernel and the system utilities. • Syslog has two important functions: • to liberate programmer from the tedious mechanics of writing log files • and to put administrators in control of logging Chapter 11 - Syslog and Log Files

19. 5. syslog: The System Event Logger • Syslog is quite flexible. • It allows messages to be sorted by their source and importance • and routed to a variety of destinations • log files, • user terminals • other machines Chapter 11 - Syslog and Log Files

20. 5. syslog: The System Event Logger • syslog consists of three parts: • syslogd • the logging daemon (config file is /etc/syslog-conf) • openlog (et al.) • library routines that submit messages to syslogd • logger • a user-level command that submits entries from the shell Chapter 11 - Syslog and Log Files

21. 5. syslog: The System Event Logger • syslogd is started at boot time • programs that are syslog aware write log entries to a special file (sometimes called /dev/log) which is either a UNIX domain socket, a named pipe, or a STREAM module (depending on the system) • syslogd reads the messages, consults its configuration files, and dispatches each message to the appropriate destinations. Chapter 11 - Syslog and Log Files

22. 5. Syslog: The System Event Logger • Configuring syslogd • The configuration file /etc/syslog.conf controls syslogd’s behavior. • It is a text file with a relatively simple format. • Selector <Tab> action • for example: • mail.info /var/log/maillog • Selectors are of the format facility.level Chapter 11 - Syslog and Log Files

23. 5. Syslog: The System Event Logger Chapter 11 - Syslog and Log Files

24. 5. Syslog: The System Event Logger Chapter 11 - Syslog and Log Files

25. 5. Syslog: The System Event Logger Chapter 11 - Syslog and Log Files

26. 5. Syslog: The System Event Logger Chapter 11 - Syslog and Log Files

27. 6. Condensing Log Files to useful information • Syslog is great for sorting and routing log messages, but when all is said and done, the end product is still a bunch of log files. • While they may contain all kinds of useful information, those files aren’t going to come and find you when something goes wrong. • Another layer of software is needed to analyze the logs and make sure some important messages don’t get lost amid the chatter. Chapter 11 - Syslog and Log Files

28. 6. Condensing Log Files to useful information • A variety of free tools are available to fill this niche, and most of them are pretty similar: • They scan recent log entries, • match them against a database of regular expressions, • and process the important messages in some attention-getting way Chapter 11 - Syslog and Log Files

29. 6. Condensing Log Files to useful information • Tools differ primarily in their degree of flexibility and in the size of their off-the-shelf database of patterns. • Two of the more commonly used log postprocessors are • swatch - perl script - • logcheck - sh script - Chapter 11 - Syslog and Log Files

30. 6. Condensing Log Files to useful information • No matter what system you use to scan files, there are a couple of things you should be sure to look for • Most security related messages • monitor failed login attempts, su, and sudo attempts • Messages about disks that have filled up • Full disks often bring all useful work to a standstill • Messages that are repeated many times • If only to clean things up Chapter 11 - Syslog and Log Files

31. Chapter 11 - Syslog and Log Files