PI Helps Solve NT Server Performance Issues and Network Incidents

1. PI Helps Solve NT Server Performance Issues and Network Incidents Or The PI’s Have It.

2. Project Started • October 1999 - Formal Project Started • Goals • Capture NT Server Performance Data In PI. • Capture Network Performance Data In PI. • Tools • A development PI Server (from Dow Corning) • A development PI License (from OSI Software) • Multiple copies of PI Perfmon Interface (from OSI Software) • A development NetScout License (from NetScout Systems Inc.) Dow Corning Corp.

3. What NT Performance Data Is Being Captured in PI? • NT Performance Counters through the PI Perfmon interface. • We have had it running since May 17, 1999 in an un-official use as an aid to Jon Peterson Dow Corning Corp.

4. Early Findings • Memory on our PI Servers was eroding. PI Trend Available. PI Batch was the main problem. Dow Corning Corp.

5. PI-Batch ProcessMemory Leak Dow Corning Corp.

6. Early Findings • Memory on our PI Servers was eroding. PI Trend Available. PI Batch was the main problem. • PI 3.2.357.8 came out with a fix for PI Batch. (Good Job Jon.) • PINetManager had a leak. Dow Corning Corp.

7. PINetMgr had a leak. Dow Corning Corp.

8. Early Findings • Memory on our PI Servers was eroding. PI Trend Available. PI Batch was the main problem. • PI 3.2.357.8 came out with a fix for PI Batch. • PINetManager had a leak. • PI 3.2.357.17 fixed the leak. Dow Corning Corp.

9. Later Findings • PI 3.2.357.17 when run on NT 4.0 SP5 also fixed a leak in PIArchss. Dow Corning Corp.

10. PIArchss Process Before SP5 Dow Corning Corp.

11. Later Findings • PI 3.2.357.17 when run on NT 4.0 SP5 also fixed a leak in PIArchss. • Ensign Agents (another tool used in Dow Corning) can go nuts and lock up one of our NT MSSQL Servers Dow Corning Corp.

12. Ensign Agents Dow Corning Corp.

13. Later Findings • PI 3.2.357.17 when run on NT 4.0 SP5 also fixed a leak in PIArchss. • Ensign Agents (another tool used in Dow Corning) can go nuts and lock up one of our NT MSSQL Servers • Exchange Server Lost it’s STORE process. Dow Corning Corp.

14. Exchange Server Notice the LOSS then change of STORE process from Exchange Dow Corning Corp.

15. NT SystemsCurrently Being Watched! • PI Server • MSSQL Servers • Exele's – EdictVB Server • FOXBoro's NT version (Our Control System Support Group is thrilled.) • Intellution SCADA servers (Our Control System Support Group is thrilled.) • Automatic Data Collection ADC Server • Exchange Servers Dow Corning Corp.

16. Locations Being Monitored • Midland – Michigan USA • Barry Wales - UK • Chiba - Japan Dow Corning Corp.

17. What Network Data Is Being Captured In PI? • Started in October 1999. • NetScout can be configured to use MSSQL. • We added two tables to the MSSQL to create an association between NetScout counters and PI Tags. • Basic Network data is captured from our Global WAN. • We are using 1 minute summarized data from NetScout. Dow Corning Corp.

18. Impact of Network on Exchange Server Exchange Server -> Dow Corning Corp.

19. Network Benefits of PI Data • Spotted several instances of Over Utilization (Can almost predict instances of Network outages for users.) • Helped to determine a Baseline to determine the need for purchased band width and line speed. ($ can be found here.) Dow Corning Corp.

20. In Summary • We helped OSI touch up some of it's code. • We spotted some troublesome software from other vendors. • We help to trouble shoot NT Server issues and outages. • We help to baseline our Global Network. • We saved some $. Dow Corning Corp.

21. How Did We Do It? • PI Tag Configurations for PI Perfmon • PI Perfmon Interface • SQL Server Use for NetScout • PI Tag Configurations for NetScout • Lessons Learned Dow Corning Corp.

22. Lessons Learned 1 • Run PI Perfmon on the server it is monitoring. • This uses the PIAPI and Buffering to push the data to a PI Server. • Trying to PULL data uses to much network bandwidth. • Trying to PULL data leaves you vulnerable to Network Outages. • Remember we are doing this across a wide area network. Dow Corning Corp.

23. Lessons Learned 2 • Really think through the compression and exception deviation percentages. • Memory usage varies a LOT. • CPU usage varies Even More. • Regression Analysis ProcessBook Addin can save you a lot of stats time. Dow Corning Corp.

24. Lessons Learned 3 • You CAN use SPC tools on an NT Server Process. • NT Server Process should run acceptablely under the rules of SPC. • You can find out what impacts you and sends you out of control. • Alarming based on Out Of Control can be done. • Alarming based on Process Conditions is also possible. Dow Corning Corp.

25. Lessons Learned 4 • OSI Believes in Continuous Improvement. • PI Batch Improvement • PI NetMgr Process • PI Archss Process Dow Corning Corp.

26. Lessons Learned 5 • When It comes to Network Stats, It is hard to tell when you have enough. • NetScout Probes Offers a lot of data. At this point Dow Corning is only capturing the tip of the iceberg. • OSI is working with NetSout to get at much more of the data. Dow Corning Corp.

27. Lessons Learned 6 • Watch Data in Long and Short Views. • Some items have data that takes weeks to develope. • Some items are best seen in the second they occur. • Only a long term historian can give you this long and short view of events. Dow Corning Corp.

28. Get the Data Have Some Fun