Performance Testing vs. Performance Monitoring

1. Performance Testing vs. Performance Monitoring Jan Warren Enbridge April 2009

2. What we shall talk about • Some of the many terms…….. • What the two disciplines mean • Comparison and contrast • Where they overlap (or do they?) • Throw in a little ITIL • Our tools • Nirvana in each world • In a joint ideal world…….. • Conclusion • Questions

3. Top of the food chain • Performance Management • System Measurement Management • Measurement Management • System Assurance • Anything else?

4. Performance Testing • AKA Quality Assurance (or Quality Control) • Performance testing: Testing conducted to evaluate the compliance of a system or component with specified performance requirements. [IEEE] • Performance testing, unit testing, system testing, integration testing, functional testing, regression testing, load testing, stress testing, volume testing

5. Performance Monitoring • AKA Quantity Assurance? • Monitoring of the End User Experience (EUE), monitoring of system infrastructure (servers, network, switches, …..)

6. Testing Performance and accuracy Pre-production Controlled Load (forced/simulated) Agent based Agentless Network Server Capacity definition SLA baselining Recreate production issues Reproduciblemeasurement under controlled conditions Monitoring Performance and availability Production Normal load Agent based Agentless Network Server Capacity planning SLA monitoring Diagnose production issues Live real time snapshot of “in service” performance Compare and Contrast

7. Where do they overlap (or do they)? • NOT the environment • The Medium (server, network, etc.) • NOT the role of application folks • NOT in terms of load • NOT in their role i.e. Testing vs. Monitoring • Possibly the tools

8. ITIL • Performance Testing • Release Management • ICT Infrastructure Management • Deployment • Performance Monitoring • Service Delivery • Service Management • Service Level Management

9. Testing HP Quality Center Quick Test Pro LoadRunner SiteScope Diagnostics (for SAP) Monitoring Compuware ServerVantage ClientVantage ClientVantage Agentless VantageView WhatsUp Gold PRTG Scrutinizer Our tools

10. Testing Nirvana • True quality assurance—nothing goes into production until completely tested and confirmed acceptable

11. Monitoring Nirvana • Indicators warn us ahead of an issue and it is already resolved before it surfaces • Failing that, the issue surfaces, the proper folks are alerted and the issue is being worked on before the calls to the helpdesk start

12. In an ideal world…….. Tests and infrastructure set up for performance testing could be ported to production for performance monitoring…………….

13. Conclusions Do I have any? • ITIL keeps them separate • Enbridge keeps them separate • Currently, although the groups talk, nothing created in the pre-prod world can be ported to the production world • Customer expectation management vs. Customer experience management • Both groups often get forgotten during development

14. Questions?