Hands-On Lab: Drilling Down into Domino Statistics

Hands-On Lab:Drilling Down into Domino Statistics Andy Pedisich Technotics

What We’ll Cover … • Using console commands to get statistics • Calculating the space saved by using DAOS • Working with a problem server’s surging CPU • Setting up for spreadsheet based data drilling • Comparing server load with average server transactions per hour • Creating other charts using custom Statrep and spreadsheets • Wrap-up

Perpetual Statistics • Domino servers constantly generate statistics • They track data on a surprising level • On almost every aspect of server operations • Agent manager • Mail and calendaring • The server’s platform • SMTP and Notes mail • LDAP • HTTP • Network • And lots more, too

Server Statistics Are Organized Hierarchically • Stats are gathered into major categories like these • And then each one has a multitude of subcategories

Subcategories of Statistics • Here’s a snapshot from the Administrator client showing some of the statistical hierarchy • This gives you a snapshot of the stats on your server • Use Refresh to get another snapshot

Statistics Come in Basic Types • The basic types of statistics are: • Stats that never change once the server is started • Snapshot stats – reflect what’s going on right now • Cumulative stats that grow from the moment server is started • These stats are available to you for: • Your Domino servers • The platform your server is running on • Your network environment

Static Statistics • Statistics that don’t change usually represent the operating environment of the server • Server.Version.Notes = Release 8.5.3FP3 • Server.Version.OS = Windows NT 5.0 • Server.CPU.Type = Intel Pentium • Disk.D.Size = 71,847,784,448 • Mem.PhysicalRAM = 527,433,728

Amazing Detail, Yours Free! • This includes OS platform, Domino version, RAM • Lots of information about disks in use • Platform.LogicalDisk.TotalNumofDisks = 3 • Platform.LogicalDisk.2.AssignedName = E • Disk.C.Size = 80,023,715,840 • And even Network Interface Card (NIC) information • Platform.Network.1.AdapterName = Intel[R] PRO_1000 MT Server Adapter • Platform.Network.2.AdapterName = Broadcom NetXtreme Gigabit Ethernet _2 • Platform.Network.3.AdapterName = Broadcom NetXtreme Gigabit Ethernet

What Good Are These Static Stats? • Think these static stats aren’t helpful? • Guess again • They are extremely valuable • If you are collecting stats correctly from all your servers, you can take a pretty detailed server inventory • Without leaving your desk • From servers all around the world, just by looking at the data we’re going to collect in the Monitoring Results database • This database is also know by its filename: STATREP.NSF

Snapshot Statistics • Snapshot stats show what’s happening at the moment youask for them • They are changing all the time • Disk.E.Free = 18,679,414,784 • Server.Users = 280 • Mem.Free = 433,614,848 • MAIL.Waiting = 250 • The best part about this is that you get lots of Domino-related stats you wouldn’t get by looking at the operating system’s performance tools

Cumulative Stats • Some stats are cumulative • They start counting from zero when you start the server • Server.Trans.Total = 31915 • SMTP.MessagesProcessed = 966 • Stats, like averages and maximums, are calculated from the cumulative ones • Server.Users.Peak.Time = 02/21/2006 07:50:33 MST • Platform.Memory.PagesPerSec.Peak = 1,364.1

Resetting Statistics • Some of these cumulative stats can be reset using the following console command: • Set Statistics statisticname • You can’t use wildcards (*) with this argument! • Here’s an example of why you might want to reset a stat: • Set Stat Server.Trans.Total • Resets the Server.Trans.Total statistic to 0 • You might want to reset this stat if: • You are starting to benchmark a new application • You are debugging an agent and want to see if it is more efficient after changes to its design

Platform Stats, Too • Platform stats vary widely from OS to OS • Getting platform stats from within Notes has great value • Track Domino server performance on an OS level even if your servers run on a variety of operating systems • For example, it’s very common to have a mix of AIX and Wintel servers • In a few minutes we’ll be discussing threshold tracking • You’ll be able to set notification thresholds universally from within Notes to track these platform stats

Getting to Platform Statistics • Domino releases 6, 7, and 8 track platform stats automatically • In earlier versions they had to be explicitly enabled and many times were disabled due to problems with servers crashing • These problems are gone • To see all platform stats – enter this console command • Show stat platform

See Server Statistics • Quickest way to see all server stats is to enter console command: • Show stat • Any place you can get to a console, you can access stats that can tell you a lot about the current state of the server • A SHOW STAT command gives you every statistic the Domino server has • Several hundred of them! • That’s really too many to deal with at once

Can I See That in a Smaller Size? • Get a better view of the stats showing just what you’re looking for using the asterisk wildcard • You can ask directly for the top level of the hierarchy • Show stat server • That shows all of the stat hierarchy under “server”

You Might Want Only Part of the Data • To get a select list of just the stats under the top-level requires the use of wildcards in your console commands • If you only want Server.Users hierarchy, use the global “*” • Show stat server.users.*

Let’s Have Some Fun With a Spreadsheet and DAOS • Domino Attachment and Object Service can save you disk space • When using DAOS, Domino saves attachments in an internal repository • If any other document in any other file or files have that same document, Domino saves a reference to the file • Attachments are only stored once on the server even though they might be used in hundreds of databases • That’s why DAOS saves disk space • But now that you’ve implemented DAOS, how much disk space have you saved • Wouldn’t you like to know how much precious disk space you’ve saved?

How to Calculate DAOS Disk Space Savings • Your DAOS data must be on a separate drive outside of the data directory of the server to use this formula • Here’s the formula to determine DAOS disk space savings • DAOS Savings =Logical Size of all databases – (Physical size of data directory + Physical size of DAOS directory) • You can get all three of these statistics using the Notes Administrator client • You have to dig a little to get them, but they are all there

Using the Notes Administrator Client to Get These Stats • We need three things: • Logical size of all databases • Physical size of data directory • Physical size of DAOS directory • Go to the Notes Administrator client and select the Files tab • The file columns will list the logical and physical size on a per-file basis • But how can you get the totals for all the files?

Getting a List of All the Files • In the Notes Administrator Client using the Files tab, click on the All button on the top right of the screen • This will change the way the files on the server are displayed • You’ll no longer see folders

The Tree View Changes into a List of Every File • You’ll see a complete list of all files on the server • But you won’t see the sum of each column • This data needs to be taken into a spreadsheet for further analysis • Note in the screen shot that every directory is listed

Select All • Use the Edit menu to select all records being displayed on the Notes Administrator Client • Or you can select one of the records and do a Ctrl A • All the records will be selected

Copy the Records to the Clipboard • Use the Edit  Copy menu or do a Ctrl C on the keyboard to copy all the records • Then open your spreadsheet

Paste into Your Spreadsheet • Use the Edit  Paste menu sequence or a Ctrl V to paste the contents of your clipboard • The contents of the File tab which listed the disk stats for all of the files on your server will be pasted into the spreadsheet • Now we can use formulas to do some calculations

You Might Want to Widen a Few Columns • Widen the columns for Logical Size and Physical Size just to see what you are working with • Remember, you need the total Logical Size, which is how big the databases would be if all the attachments were contained in the database • And you need the total Physical Size, which is how big the databases actually are

Add @Sum Formulas to Logical and Physical Columns • Use an @sum formula to total the disk size at the bottom of the logical size and physical size columns • You can do this also by going to the bottom of each of the columns and clicking the autosum formula option • This is an Office 2007 screen shot • You now have 2 of the 3 stats you need

You Need the Size of the DAOS Storage • To get the size of the DAOS storage, you have to query the statistics on the server • Do this by entering the following using the Notes administrator client server console • Show stat disk.* • Among the disk statistics will be the free space and size of the disk that houses the DAOS store • Copy them to the clipboard

Using the Excel Import Wizard • If you are using Excel 2007 or better you can use Paste Special, and use the text import wizard to allow the disk size values to be pasted into cell • Otherwise you will have to copy the numbers out of the line of statistics you pasted in

Paste into the Spreadsheet • Paste them into the bottom of the spreadsheet • Use a formula to subtract the DAOS free space from the total DAOS space allocated • That will give you the space used by DAOS

Create the Final Formula • Create a formula • DAOS Savings = Logical Size of all databases – (Physical size of data directory + Physical size of DAOS directory) • It’s something great to show management

Dealing with Problematic Servers • Sometimes there are servers with issues that crop up • We would like to collect statistics for analysis from these systems more frequently than we do from the standard statistics collection interval • If you try to add a second collection interval on a server you’ll get this:

Each Server Is Allowed to Collect Stats with Only One Interval • A server can only have one collection interval • You must create a 2nd collection document for another server • Don’t forget to add the “collect” task to servertasks= in NOTES.INI • Let’s look at a server that has CPU spikes • First we create a statistics collection document for a second server to take statistics from our problem server

Set the Collection Interval for Five Minutes • Set collection interval for 5 minutes • Do not check any filters!!! • They tell the collector to ignore the statistics you checked • Note that stats are being logged to a database called ProblemServer.NSF • Used exclusively to track CPU util of Traveler task • Note that the data in this example has been fictionalized for effect

Create a Special View That Tracks CPU Utilization for Traveler • In this case it’s the Traveler CPU we want to track • We create a custom view for the collecting database that only has the server name, the time of collection, and the statistic called Platform.Process.Traveler.1.PctCpuUtil • This will be used to easily create a graph of the CPU activity

Collect the Data, Copy It as a Table from the Custom View • After collecting a week’s worth of data, we experience the CPU utilization • All the data in the view is selected using Ctrl A • It is copied as a table • Copying views as a table is my favorite feature in Notes • A Monitoring Results template is posted on my web site • A URL to this template is included at the end of the presentation

Data Has Been Copied to a Spreadsheet • A simple paste of the data puts it into a spreadsheet where we are ready to turn it into a chart

Use the Tools in Your Spreadsheet to Create a Graph • Select the columns Collection Time and Traveler CPU • Create a graph from the data • In this example, a scatter chart type with smooth lines is being used

The Resulting Graph • This produces an excellent graph of the CPU utilization over a ten day period with samples being taken at intervals of 5 minutes • And it took less than 5 minutes to make this chart • One adjustment was made to the x-axis formatting and the legend was removed

The Statrep Template’s Only Export View • The default Lotus Statrep template’s Spreadsheet Export view just doesn’t seem to give us enough power • Pulling the data into Excel, then analyzing and graphing the data can often give you amazing insight into usage patterns • This information will be invaluable when: • Trying to consolidate servers • Troubleshooting performance issues

Everything Is Everywhere • Keep in mind that every single statistic that is generated is contained in every document in the Monitoring Results database • You just can’t see it all because it’s not in views or documents • And views are the most important place to have it because that’s where it gives you the ability to compare samples • And analyze trends

The Stats Are There, Now You See Them • There is a new, customized version of the Monitoring Results database • TechnoticsR85Statrep.ntf • It has all the views that are on the original Statrep • Plus over a dozen additional views to help you analyze the stats your servers generate • You can download this from my blog • www.andypedisich.com • Look for the link to Admin2013 resources

Analysis Tools • Let’s cover the basics of the Statrep views used in the data export process • And the special Excel spreadsheet that contains custom formulas

You Need a Better View of the Situation • The data export views are designed to be exported as CSV files • Each has key fields that are important to the export • Hour and Day generate an integer that represents the hour of the day and a day of the week • Hour 15 = 3:00 PM • Day 1 = Sunday, Day 7 = Saturday • These are used in hourly and daily calculations in pivot tables

Export Views Are All Flat Views • Any view that is used for exporting data is flat, not categorized • This makes it easier to manipulate in pivot tables in Excel • There are columns in the export views that appear to have no data • They will be filled with a formula when brought into Excel

Formulas Are Already Available • There is a spreadsheet containing my formulas to help you develop charts for all of this data • It’s available on SocialBizUG.org • Master Formula XLS Stat Exports- Technotics -V 2-4.xls • The views and spreadsheet will all fit together in a few moments

Hands-On Lab: Drilling Down into Domino Statistics