Automated Capacity Planning and Support for Enterprise Applications

1. Automated Capacity Planning and Support for Enterprise Applications Dharmesh Thakkar School of Computing, Queen’s University, Canada

2. Performance Activitiesin The Waterfall SDLC Verification: Performance test need to be conducted to verify that the required performance objectives are met

3. Performance Activitiesin SDLC Maintenance: Continuous monitoring, capacity planning and tuning need to be done to ensure that the system continues to meet the performance objectives

4. Overview of Thesis Challenge: The goal of performance testing is not to find bugs, but to establish benchmarks Build a capacity calculator based on the performance model Challenge: Load test and stress test workloads need to be compared against the customer workloads Modify the workload or the deployment scenario to put excessive stress on one or more system resources The capacity calculator helps customer determine required hardware resources to support the anticipated workload Run the application for long periods under high workloads • Customers engage the vendor’s support analyst to resolve the issues with system • Challenge: The support analyst needs to troubleshoot the customer deployment • Performance issues arise in customers’ production environment, which need troubleshooting

5. Overview of Thesis:Our Contributions A framework for building performance model based capacity calculator Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ. Acceptance rate: 38% A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China. Acceptance rate: 26%

6. Verification Phase:An Example of A Capacity Calculator

7. Building a Measurement based Capacity Calculator

8. Performance Modeling Challenges • Large number of tests to be executed • Limited time for performance modeling • Manual process of setup, execution and analysis • Risk of error and repeating the full modeling process

9. Addressing the Challenges of Performance Modeling • We created the framework to: • Automate the process • Speed up the process • Focus research on specific areas

10. Framework for Building Model based Capacity Calculator • Case Studies: • Dell DVD Store • A large enterprise application

11. Customization Cost

12. Maintenance Phase:Customer Support

13. Retrieving Relevant Reports • State of Practice: • No systematic techniques to retrieve and use information for future engagements • Keyword searching is limited: • depends on the search skills and experience of the analyst and peculiarity of the problem

14. Customer SupportProblem Statement • We want to find customers with similar operational and problem profiles • We can reuse prior solutions and knowledge Other Customers • Heavy Email, Light Web, Light calendar • Light Email, Light Web, Light calendar • Light Email, Heavy Web, Light calendar • Heavy Email, Heavy Web, No calendar • Light Email, Light Web, Heavy calendar New Customer Engagement Compare

15. Using Logs forCustomer Support • Execution logs are readily available and contain • Operational Profile: usage patterns (heavy users of email from device, or to device, or light users of calendar, etc.) • Signature Profile: specific error line patterns (connection timeout, database limits, messages queued up, etc.) • Find the most similar profile

16. Execution Logs • Contain time-stamped sequence ofevents at runtime • Readily available representatives of both feature executions and problems <time> Queuing new mail msgid=ABC threadid=XYZ<time> Instant message. Sending packet to client msgid=ABC threadid=XYZ<time> New meeting request msgid=ABC threadid=XYZ<time> Client established IMAP session emailid=ABC threadid=XYZ<time> Client disconnected. Cannot deliver msgid=ABC threadid=XYZ<time> New contact in address book emailid=ABC threadid=XYZ<time> User initiated appointment deletion emailid=ABC threadid=XYZ

17. Example Other Customers Compare

18. Technique to Retrieve Relevant Engagement Reports

19. Case Studies • Case Study I • Dell DVD Store open source application • Code instrumentation done for event logging • Built the execution log repository by applying synthetic workloads, changing the workload parameters each time • Case Study II • Globally deployed commercial application • More than 500 unique execution events

20. Case Study Results • Dell DVD Store • 100% precision and recall on both operational profile based and signature profile based retrieval • Commercial Application • 100% precision and recall for signature profile based retrieval • Results for operational profile based retrieval:

21. Sources of Errors • Events that do not correspond directly to a particular operational feature, such as idle time events, server health check events, startup and shutdown events • Imbalance in the event logging

22. Summary:Our Contributions A framework for building performance model based capacity calculator Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ. A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China.