Admin Aspirin — Reduce Performance Headaches Through Proper Domain Monitoring

Admin Aspirin — Reduce Performance Headaches Through Proper Domain Monitoring Andy Pedisich Technotics

In This Session ... • Let’s talk about your servers • Are they struggling to keep up or not even breaking a sweat? • Do you have too many users on a server or could you easily consolidate the ones you have? • Are they having problems, but you’re just waiting for a help desk ticket to come in before you find out what’s wrong? • This session takes you behind the scenes to show you how you can be a more powerful administrator using data that’s always there, but a little hard to find

What We’ll Cover … • Understanding the fundamentals of statistic collection • Exploring the native Notes monitoring system • Taking advantage of server activity logging and trending • Exposing message tracking’s capabilities • Using the Domino Configuration Tuner • Dominating your domain with DDM • Customizing STATREP.NSF to pull out hidden data • Analyzing statistics for proactive problem solving • 10 statistics to monitor for top performance and stability • Wrap-up

The Two Things Needed • Two things are required for statistics collection: • The Collect task must be running on any server that is designated to collect the statistics • Not all servers should run the Collect task • The EVENTS4 database must have at least one Statistics Collection document • Statistics should be collected centrally on one or two servers so that the data is easy to get to • Stats should be collected every hour to be effective • EVENTS4 should be the same replica on all servers in the domain

We Know What the Replica ID Should Be for EVENTS4 • Replica ID of system databases, such as EVENTS4.NSF, are derived from the replica ID of the address book Database Replica ID NAMES.NSF 852564AC:004EBCCF CATALOG.NSF 852564AC:014EBCCF EVENTS4.NSF 852564AC:024EBCCF ADMIN4.NSF 852564AC:034EBCCF • Notice that the first two numbers after the colon for the EVENTS4.NSF replica are 02 • Make sure that EVENTS4.NSF is the same replica ID throughout the domain by opening it and putting it on your desktop

Want to Add Every EVENTS4.NSF to Your Desktop? • Add this code to a button on your toolbar • This is courtesy of Thomas Bahn • www.assono.de/blog • This code will prompt you to pick the servers that have the database you want on your desktop • Then it will prompt for the name of the database • And open it on all the servers you’ve selected _names := @Subset(@MailDbName; 1) : "names.nsf"; _servers := @PickList([Custom]; _names; "Servers"; "Select servers"; "Select servers to add database from"; 3); _db := @Prompt([OkCancelEdit]; "Enter database"; "Enter the file name and path of the database to add."; "log.nsf");@For( n := 1; n <= @Elements(_servers); n := n + 1;@Command([AddDatabase]; _servers[n] : _db) )

A Required Design, but No Required Name • There has to be a STATREP.NSF on every server • It is used by the server to store monitoring data • It must be designed using the Statrep5.ntf Monitoring Results template • Its default title is Monitoring Results • But you don’t have to use one of those for your statistic collection repository • Create your own collection points and give the database a unique name

Event Monitoring Details • Event monitors are set in the EVENTS4 database • Event generators • Specify what built-in event shouldbe monitored • Event handlers • Specify the action that Dominotakes when an event generator occurs • Here’s the high-level concept of monitoring: • Setup an event generator to capture an important moment • Use an event handler to be notified about it • You can also use event handlers to send you a notification of anything that appears on the console or in a LOG.NSF

Event Generators Are Your Friends • Event generators are pre-set groups of categories to monitor • Some of the more useful are: • Database generators • Triggers when the ACL in the address book is changed • Statistical generators • Trigger for low disk space, too many messages waiting • Domino server response generator • Triggers when servers are not reachable • Task status generator • Triggers when HTTP task is down

An Example: Database Event Generator • The ACL of NAMES.NSF should be monitored for changes in every Notes domain • Once properly set, the ACL of NAMES.NSF should rarely change! • All kinds of bells and whistles should go off when it does • We’ll talk about notification in a moment • Here’s how to set up the monitoring of the ACL • Select New Database Event Generator

Setting Up the Database Generator • Select Names.nsf • You can choose either a single server, such as the administration server for the address book, OR • All servers in the domain • I like to pick all servers in the domain • Admins won’t get away with anything! • But I do get a storm of messages when an ACL change occurs • Every server tells me aboutthe change

Another Event Generator Example • Domino Server Response Event Generator • Checks connectivity/port status of server’s network • One server checks others by sending a probe • It’s a good idea to try opening Names.nsf • If you can’t open Names.nsf, then something is wrong!

Event Handlers Set Up Notification • When configuring an event generator, you can click the button on the final tab for the Event Handler Wizard • Simple to do and guarantees it will work the way you want it to • The wizard will walk you through the process of creating an event notification method for the event generator you just made

Turning on Activity Logging • Activity logging is switched on in server configuration documents • Select all the server tasks you would like to use to produce activity logging data • If you want to see activity trends, enable all tasks except Domino.MAIL • At a minimum, you must enable Domino.Notes.Session and Domino.Notes.Database

Check Points and Prime Shift • Leave the default checkpoint interval at 15 minutes • It seems to be sufficient • We’re surprised some of these are checkboxes since you really need them all for activity logging • Log checkpoint at midnight is required for both activity logging and activity trends • Log checkpoint for prime shift is required for activity trends • Enter whatever you consider as your prime shift • And voila! Activity logging is started!

Activity Logging • Activity logging causes Domino to write information to the LOG.NSF about the tasks you are interested in • Expect the log to grow to about 2 GB • The data is kept in a few hidden views in a form that is not meant for humans to see • IT is waiting there for us to access in two ways: • Activity analysis • Activity trending

Activity Trends Are Very Useful • You can enable the activity trends collector in the same configuration document that you used to set up activity logging • If you activate it in the default config document it creates an ACTIVITY.NSF database on every server in the domain • Then it collects data from the log every night at 3:23 AM • You can run the task manually at the console using Load trends

Activity Data Rocks! • Activity trending gives you an unbelievable amount of data about your environment such as: • How your protocols are used • How your servers are utilized during prime time • Growth rates for databases

Inactivity Is Important Too • Inactivity is also tracked • Lets you rid your systems of dead mail files and applications that are truly no longer being used • A little bit of investigation using Activity.nsf will probably save you gigabytes of disk space • And you won’t be wasting time backing up obsolete databases

But Wait, There’s More … • Activity trending also provides information about user activity • It’s probably more information than you need for everyday administration • But it’s great for capacity planning or when you have a server consolidation to accomplish

Catalog Must Be Enabled • Make sure there are no errors when the activity trends are rolled out • Check the Run Log for details about the collection process • The catalog task must run on each server you want to include in activity trending • You’ll see an error message in the Run Log if there is no catalog on the server

Demo

Message Tracking • Allows administrators to track all messages • Users can track their own messages • Shows delivery status and current disposition • Read/Unread, Deleted • Configurable • Track message subject • Don’t track messages from certain users • Great tool to use when users complain of “lost” email!

Enabling Message Tracking • From the Server Configuration document  Configuration tab in the Administration client • Go to the Router/SMTP – Message Tracking tab • Complete fields as needed • Tracking Interval controls how often data is written to the message tracking database • The default interval is 15 minutes, which is a great place to start

Access to Message Tracking • The Access Settings fields are used to limit who can track other users’ messages • Include administrators in this field • Maybe Help Desk? • Make sure that LocalDomainServers is included if you need to track across servers

Message Tracking Reports • Daily/weekly/monthly options for all reports: • Message volume • Largest messages • Top 25 next hops • Top senders and receivers • By count • By size • Message status • Delivered/not delivered/in process

Caution Using MTC on iSeries • Servers have seen the MTC task take up large amounts of CPU • Up to >50% on iSeries systems • Appears to be related to full text indexing of the MTSTORE.NSF • To remediate this problem • Stop MTC task • Delete and recreate FTI on mtdata/mtstore.nsf • Start MTC task • This can be done on all platforms • I’ve seen the problem on Wintel and UNIX

Domino Configuration Tuner • Domino Configuration Tuner (DCT) produces a list of items scanned and provides complete explanations of each item • It’s a fast and easy way to take stock of your domain

Domino Configuration Tuner (cont.) • Domino Configuration Tuner (DCT) compares your server settings to an IBM catalog of Best Practices • DCT looks at settings in the Domino Server documents, the NOTES.INI file, and advanced database properties • It also looks for worst practices • All servers in your domain can be evaluated at once

Batteries Not Included • The DCT is not included with the Domino install files • You can download the latest and greatest version here: • http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg24019358 • IBM updates the design and rules that the application uses on a regular basis • Make sure you use the Check for Updates before you scan your systems

Run Scan and Wait for the Results • Clicking on Run New Scan will start the tuner which will access your domain’s address book to pull out a list of servers • Select the servers you want the tool to scan • You’ll also have the opportunity to manually add servers and optionally create a name for the scanning • This is a very valuable free asset from IBM

DDM Details • DDM provides a single location where administrators can access issues that are affecting multiple servers and databases • Reduces the risk of unplanned downtime • The DDM database is the central repository of all monitoring data • This can be data collected by probes that you can configure • Result messages from event generators that you configured in previous releases of Notes • Results for routine checks that run as part of specific server tasks, such as the router or replicator

Configure DDM for Centralized Data Collection • DDM.NSF has the most value when it is the central repository for all issues • It will contain all of the issues that come from all of the servers • There is no collection hierarchy set up by default • If your collection hierarchy looks like the screen shot below, the collection hierarchy has not been configured yet

Aggregate Data Centrally • A DDM server collection hierarchy lets you aggregate the data onto a key server or servers • This must be configured in the EVENTS4.NSF • The simplest hierarchy is to configure one server to collect from all servers in the domain

Rolling Up the Data • DDM data roll-up propagates the probe results up the DDM server collection hierarchy • Data roll-up is accomplished using Domino’s selective replication to transport the data • The replication formulas are created automatically when you define your DDM server collection hierarchy • Collection replication occurs about every five minutes • It’s hard coded and can’t be changed

Address Issues by Severity Level • Looking at issues by severity gives you the chance to deal with the most important issues first • They are broken out by severity category

Looking at a Failure • This simple event in DDM tells us what the problem was and when it occurred • And it offers a possible solution • This example shows the basics of DDM

Analysis Probes • Configure DDM using probe documents in EVENTS4.NSF • Otherwise known as the Monitoring Configuration database • A probe is a check, or set of checks, configured to run against one or more servers, databases, and services • A probe returns status and results to the Domino Domain Monitoring Database – DDM.NSF • The DDM.NSF is created automatically when an R7/R8 server starts • You can create multiple probes for each feature area • Just remember that you don’t need probes to use DDM!

What’s in These Probe Documents? • Probe type and probe subtype • For example, Security is a probe type • One of its probe subtypes is Best Practices • This combination of probe type and probe subtype creates a Security probe

Extra Information About the Probe Is Provided • These probe documents also contain a general description of the probe, its purpose, and its intended use

Demo of Domino Domain Monitoring

Plenty of Cool Probes • R8 gives us 58 default DDM probes to work with • R7 gives us 48 – still plenty to get us started • You can get probing as soon as R7/R8 is up • Just plug in your server info to get DDM started • You can also create new probe documents • Define and customize your own probes

Zeroing in on Probes • We’re going to focus on three probes that have a high value in almost every Domino domain: • Application probes • Web probes • Security probes

Agents Are Tracked by Application Probes • Application probes monitor agents • None of these probes can be scheduled • They are all real-time monitors • Application probes have the following subtypes: • Agents behind schedule • Agents ranked by CPU usage • Agents ranked by memory usage • Long-running agents

Web Probes • Web probes provide results as a Domino Event document in the DDM database • There are two subtypes: • Configuration probe • Verifies the accuracy of a Web server’s configuration • Best Practices probe • Compares servers’ configurations against a known good server • The default schedule for these probes is once a month

Admin Aspirin — Reduce Performance Headaches Through Proper Domain Monitoring