New SA Training Topic 9: Logging, Monitoring, and Performance

1. Logging Windows – “Auditing” Linux – syslog Monitoring MRTG Big Brother Performance Perfmon Sysstat New SA TrainingTopic 9: Logging, Monitoring, and Performance

2. Logging • Windows • Logging is usually dealt with as “Auditing”, this information is reviewable in the Event Viewer • Many items are NOT audited by default, this must be enabled • Logging is often incomplete as compared to that done by Linux • IIS logs to %systemdir%\system32\LogFiles\ • These can be set to store elsewhere though • The content and form is to some degree configurable

3. Logging (cont.) • Linux • Syslog – syslogd collects messages from processes and routes them as needed • Syslog.conf entry format • facility.level action • Facilities – kern, user, lpr, daemon, auth, authpriv, mail, cron, syslog, mark, local* (0-7) • Levels – emerg, alert, crit, err, warning, notice, info, debug, none, mark (selected or higher) • Action – write to file (often /var/log/messages), message a user (or list of users), syslog on another host (@host.domain.net), etc.

4. Logging (cont.) # Sample syslog.conf – This prints most sys. events to the console, # emergencies to everyone, alerts to root, and auth.info and all # warnings to otherhost kern.warn;*.err;authpriv.none /dev/console *.emerg * *.alert root auth.info;*.warning @otherhost.domain.net # send mail and kernel/firewall messages to their respective logfiles mail.* /var/log/mail kern.* /var/log/kernel_n_firewall # operators: “=“ only this, “!=“ all but this, “!” log levels below kern.=alert /var/log/kernel_alerts # save the rest in one file, but exclude mail from these (.none) *.*;mail.none /var/log/messages

5. Monitoring • A service is not in production until it's monitored • Level 1 - would include things like viewing Windows processes, Linux “top” command, netstat (both platforms), etc. • Level 2 - might be packet sniffers such as tcpdump and Ethereal • Level 3 - might included SNMP based utilities • You should already be familiar with the first two levels

6. MRTG • Multi Router Traffic Grapher (MRTG) • A tool to monitor the traffic load on network-links • Runs from cron (Windows - scheduler?) • Generates HTML pages and images that provide a LIVE visual representation of this traffic • Based on Perl and C and works under UNIX and Windows NT • RRDtool • New product by same author • Improved data consolidation and graphing • Needs additional software to collect data (Cricket)

7. MRTG (cont.) • GREEN ###Incoming Traffic in Bits per Second • BLUE ###Outgoing Traffic in Bits per Second

8. MRTG (cont.) • GREEN ###Incoming Traffic in Bits per Second • BLUE ###Outgoing Traffic in Bits per Second

9. Big Brother • Monitors System and Network-delivered services for availability • An almost real-time indication of network status is displayed on a color-coded web page • Can handle notification via E-Mail, pager, or text messaging

10. Big Brother (cont.)

11. Performance • Performance is a huge topic • Three step cyclic process of managing performance • Measure - determine current performance levels • Estimate - required/best case performance levels • Tune system - to meet requirements/best case levels

12. Perfmon • Lets you keep an eye on just about anything • Things are grouped into “objects” and objects are divided into “counters” • Example: things related to the CPUs are in an object called “processor” • Lets you either log info long-term or view in real time • Start/run/perfmon • Start/Programs/Admin Tools/Performance

13. Perfmon (cont.)

14. Perfmon (cont.) • Counter logs • Create a log based on Objects and/or Counters (same items viewable in Performance Monitor) • Store the collected information for later viewing and evaluation • Trace logs • Event Tracing for Windows (ETW) is a tool for performance-testing and diagnostics • Gives developers a mechanism with which to determine their applications' performance effects on Windows Server 2003, Windows XP, and Windows 2000 platforms • Administrators can use ETW to find out what's happening in their internal Windows systems, Microsoft applications (e.g., Microsoft IIS), and third-party applications and troubleshoot any problems they might find • ETW can also help administrators with capacity planning by letting them monitor a system under real workloads to see how it performs for a given set of transactions

15. Perfmon (cont.) • Alerts • Give notice when “something” happens • Can alert by • Adding log entries • Network messaging • Running a program • Some examples • free disk space (logical disk/free megabytes) • general network congestion (network percent network utilization) – requires Monitor Agent • logon attempts for ftp or http servers • logon errors (Server/errors logon)

16. sysstat • A set of commands for Linux • sar - collects and reports system activity information • The information collected by sar can be saved in a file in a binary format for future inspection • The statistics reported include I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization, kernel activities and TTY statistics, etc. • sadf - used to display data collected by sar in various formats (XML, database-friendly, etc.) • iostat - reports CPU utilization and I/O statistics for disks • mpstat - reports global and per-processor statistics • Both single and mulit-processor machines are fully supported

17. sysstat (cont.) • Apple’s Dashboard Widget for Sysstat