The University of Akron Summit College Business Technology Dept.

1. The University of AkronSummit CollegeBusiness Technology Dept. 2440: 141Web Site Administration Web Server Monitoring and Analysis Instructor: Enoch E. Damson

2. Monitoring and Analyzing Systems • Monitoring operating systems, Web servers, applications, etc typically involves analyzing log files • Log files – contain information recorded by the operating system in response to certain events Monitoring and Analyzing the Web Server Environment

3. Monitoring Operating Systems • Logs are used to detect problems • OS, application, or security problems • Various tools can monitor performance Monitoring and Analyzing the Web Server Environment

4. Monitoring Windows • Performance monitoring allows you to compare system performance over time • Windows Task Manager highlights CPU and memory usage • You can modify services to notify you if a service fails Monitoring and Analyzing the Web Server Environment

5. Windows Event Viewer • The event viewer contains six event types shown in the left pane Monitoring and Analyzing the Web Server Environment

6. Windows Event Logs • System and application events display three levels of messages • Information • Warning • Error • Because many messages can be generated, a filter focuses on what you want to see • Over time, the logs fill up so you should clear them or save them Monitoring and Analyzing the Web Server Environment

7. Monitoring Linux • Logging is controlled by the syslogd daemon • Below are some facilities which represent daemons using syslogd Monitoring and Analyzing the Web Server Environment

8. Eight Levels of Message Priorities in syslogd Monitoring and Analyzing the Web Server Environment

9. Web Server Log Files • Files that keep track of Web server transactions • Most Web servers write two log files to disk: • Access log – contains a line for each Web server request • Error log – contains a line for each generated error response • When log files grow: • A common practice is to put the log files on a separate drive or partition • A better solution is to rotate the log files • Rename or remove the log files at regular intervals (weekly, monthly, etc) Monitoring and Analyzing the Web Server Environment

10. Web Server Log File Formats • Most Web servers support at least two logging formats: • Common Logfile Format (CLF) • Extended Logfile Format (ELF) • Most Web servers also allow the administrator to specify a custom format, along with the above formats • A standard logfile format makes it easier for users to understand files from different servers • Allows third-party logfile analysis tools to support many different Web servers Monitoring and Analyzing the Web Server Environment

11. Common Logfile Format (CLF) • The NCSA and CERN Web servers first used this file format • Many Web servers now support this format (IIS, Apache, Netscape Enterprise, etc) • Each line in the file represents a unique request • Has a fixed format with seven fields to be logged: • remotehost • rfc1413 • authuser • [date] • “request” • status • bytes Monitoring and Analyzing the Web Server Environment

12. Common Logfile Format… • remotehost – remote (client) hostname or IP number • rfc1413 – remote username • rfc1413 defines a protocol used to determine the identity of a client that requests a resource from the server • Seldom used on Internet servers because it slows the server’s response • A “-” is entered into the log if the server is unable to determine a userid • authuser – when required, the username by which the user has authenticated is provided • A “-” is used for normal unrestricted requests • [date] – date and time of the request • Enclosed in brackets for potential spaces • “request” – HTTP request line exactly as it came from the client • Enclosed in quotes for potential spaces • status – HTTP status code returned to the client • bytes – content length of document transferred • Example: 127.0.0.1 - - [24/Oct/2006:09:11:55 -0500] "GET /test.asp HTTP/1.1" 200 626 Monitoring and Analyzing the Web Server Environment

13. Extended Logfile Format (ELF) • Used to log more information or omit certain fields • Allows the administrator to specify exactly which fields to log and in what order • Each represents a request like CLFs but the beginning of the file also contains some configuration directives • Each directive line begins with a # • Two directives are required and must precede all entries in the log file: • Version – specifies the version of the ELF to use • Fields – specifies what data to record in the logfile Monitoring and Analyzing the Web Server Environment

14. Extended Logfile Format… • The fields directive here specifies 8 out of several available fields: • date – client request date • time – client request time • c-ip – client IP address • cs-method – HTTP request method • cs-uri-stem – file requested by client • sc-status – HTTP status code returned to the client • sc-bytes – number of bytes sent from server to client • cs-version – version of HTTP used by client to connect to the server Monitoring and Analyzing the Web Server Environment

15. Error Logs • Contains informational messages and debugging information • Useful for: • Finding problems with the server • Debugging server-side programs and new configurations • Most server packages allow the administrator to control what types of messages are logged to the error log file • The format is usually not configurable like ELFs but allows some flexibility in choosing the severity and type of messages to log • E.g only critical messages may be logged if a server is running smoothly Monitoring and Analyzing the Web Server Environment

16. Referrers • Determines what Web page was used by the client to access a server • May be the URL of a search engine or any Web site with a link to the Web server • A “-” is used if there was no Referrer header sent • The Referrer header is not sent in the following circumstances: • The users enters the URL by hand • The user clicked on a link to regular file and not a Web page on a public site • The user loaded the URL from a bookmark file • The Referrer URL is on a private (internal) Web site • The user or browser has disabled sending the Referrer header Monitoring and Analyzing the Web Server Environment

17. Monitoring IIS • IIS has specific counters for use in the Performance Monitor • The System event viewer provides specific information • IIS has extensive logging capabilities • There are default log formats used by various third-party applications that analyze logs Monitoring and Analyzing the Web Server Environment

18. Monitoring ApacheError Logs • By default, syslogd sends Apache messages to /var/log/boot.log • Location of the error log • ErrorLog logs/error_log • Logs refers to /var/log/httpd • You can create a different error log for each virtual host Monitoring and Analyzing the Web Server Environment

19. Monitoring ApacheTransfer Logs • Transfer logs tell you about the use of your Web site • The default log is based on a combined format • Determined by the CustomLog directive in the configuration file (httpd.conf) • There are a number of sample formats • By default, logs are stored in /var/log/httpd/access_log Monitoring and Analyzing the Web Server Environment

20. Monitoring DNS • BIND uses a logging statement that you configure in named.conf • BIND defines logging in two parts: • Channel defines where logging is sent • Category defines what will be sent • If the channel is going to a file, use the versions option to define the number of backups • Size option sets maximum size of the file • print-time adds the date and time to the file Monitoring and Analyzing the Web Server Environment

21. BIND Categories Monitoring and Analyzing the Web Server Environment

22. Monitoring Exchange Server • Exchange server uses the application portion of Event viewer • You can enable four types of logs • audit – access to mailboxes • protocol – commands used for SMTP, etc • message tracking – senders and receivers • diagnostic – analyze detailed problems Monitoring and Analyzing the Web Server Environment

23. Analysis Tools for the Web Server • Analysis tools extract system data from logs and format the data • For IIS, one of the popular tools is WebTrends • Helps you determine the source of Web traffic • Determines which pages are most popular • Several different reports • 123LogAnalyzer is available for both IIS and Apache • Many reports are similar to WebTrends Monitoring and Analyzing the Web Server Environment

24. Log File Analysis • Simply looking at log files can provide a lot of information about activities or requests on a server • Simply counting the number of lines in an access log file can help determine the number of hits • Log files may be reviewed regularly to find the common errors logged • Some of the common errors include: • Dead links • Requests for non-existing files • CGI scripts not working properly • Permissions problems • Some of the open-source log analyzers are: • Analog (http://www.analog.cx) • Webalizer (http://www.mrunnix.net/webalizer) • Report Magic (http://www.reportmagic.org) Monitoring and Analyzing the Web Server Environment

25. Statistics • With the help of several log analyzer programs, some of the statistical information that can be extracted include: • Most requested pages • Top entry pages (the first page clients enter a site through) • Most used browsers • Bandwidth utilization • Most active domains • Top referring sites and URLs • Error counts • Information about search engines (most common search engines, common queries, etc) • Some of the widely used commercial log analyzer products include: • WebTrends (http://www.webtrends.com) • Wusage (http://www.boutell.com/wusage) • A database could also be used to store log information to increase efficiency of logging and report generation • Not all Web servers support logging to a database Monitoring and Analyzing the Web Server Environment