1 / 26

Software Performance Testing Based on Workload Characterization

Software Performance Testing Based on Workload Characterization. Elaine Weyuker Alberto Avritzer Joe Kondek Danielle Liu AT&T Labs. Workload Characterization. A probability distribution associated with the input domain that describes how the system is used when it is operational

caesar-browning
Download Presentation

Software Performance Testing Based on Workload Characterization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Software Performance Testing Based on Workload Characterization Elaine Weyuker Alberto Avritzer Joe Kondek Danielle Liu AT&T Labs

  2. Workload Characterization A probability distribution associated with the input domain that describes how the system is used when it is operational in the field. Also called an operational profile or operational distribution. It is derived by monitoring field usage. We have used it to select (correctness) test cases, predict risk, assess software reliability, predict scalability, select performance test cases, among other things.

  3. Steps in Characterizing the Workload • Model the software system. • Identify key parameters that characterize the system’s behavior. • Getting the granularity right. • Collect data while the system is operational or from related operational system. • Analyze the data and determine the probability distribution.

  4. System Description Automated customer care system, built by another company, that can be accessed by both customer care agents and customers. It contains a large database with a web browser front-end and a cache facility. For this system data was collected for 2 ½ months and analyzed page hits at 15 minute intervals.

  5. Implementation Information The system was implemented as an extension to the http web server daemon, with a mutex semaphore used to implement database locking. The system is single threaded. Queued processes executed spin lock operations until the semaphore was free.

  6. Before Performance Testing Prior to doing performance testing, users were complaining about poor performance, and the database was “hanging” several times a day. The hypothesis was that these problems were capacity-related, and the vendor was contacted but unable to solve the problems.

  7. Performance Testing Goals Help them determine: • Which resources were overloaded. • Effects of the database size on performance. • Effects of single vs. multiple transactions. • Effects of the cache hit rate. • Effects of the number of http servers.

  8. System Information • The web-server database was modeled as an M/D/1 queue. • The arrival process was assumed to be Poisson. • The cache hit rate was determined to be central to the system’s performance. It ranged between 80-87%, with the average being 85%.

  9. Distribution of User Requests

  10. Computing the Cache Hit Probability

  11. System Observations Heavier workload for agents on weekdays, peak hours in the afternoon, with little day-to-day variation. Customer peak hours occurred during the evening. Little change in workload as users become familiar with the system. (Agents are already expert users and execute a well-defined process, while individual customers tend to use the system rarely and therefore also maintain the same usage pattern over time.)

  12. What We Achieved • Characterized the agent and customer workload, and used it as a basis for performance testing. • Identified performance limits and as a result detected a software bottleneck. • Provided recommendations for performance improvement. • Increased the understanding of the use of data collection for performance issues.

  13. What We Learned The system was currently running at about 20% utilization. The CISCO routers were not properly load balanced. The spin lock operations consumed CPU time which led to a steep increase in the response time. (We used the SUN SE toolkit to record the number of spin locks).

  14. No Caching 7 6 5 4 Customer Agent avg CPU cost(sec)/request 3 2 1 0 0 0.5 1 1.5 2 2.5 Load (hits/sec)

  15. No Caching 40000 35000 30000 25000 Customer avg response time(ms) 20000 Agent 15000 10000 5000 0 0 0.5 1 1.5 2 2.5 Load (hits/sec)

  16. All Requests Retrieved From Cache 5000 4500 4000 3500 3000 avg response time(ms) Customer 2500 Agent 2000 1500 1000 500 0 0 20 40 60 80 100 120 140 160 180 Load (hits/sec)

  17. Simulated 85% Cache Hit Rate 60000 50000 40000 30000 Customer avg response time (ms) Agent 20000 10000 0 0 0.5 1 1.5 2 2.5 3 3.5 4 -10000 Load (hits/sec)

  18. In Particular Delay strongly depends on caching: Found in cache ~ 100ms Retrieved from database ~ 5 secs Current Available capacity: Customer: 2 hits/sec Agent: 2.5 hits/sec Average demand: Customer: 10,000 hits/day = 0.12 hits/sec Agent: 25,000 hits/day = 0.29 hits/sec Busy Hour demand: Customer: 784 hits/hour = 0.22 hits/sec Agent 2228 hits/hour = 0.62 hits/sec

  19. Effect of Database SizeCustomer – Cache Off 60000 50000 40000 avg response time (ms) Small DB (200 MB) Medium DB (400 MB) 30000 Large DB (600 MB) 20000 10000 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Load (hits/sec)

  20. Effect of Database SizeAgent – Cache Off 35000 30000 25000 20000 Small DB (200 MB) avg response time (ms) Medium DB (400 MB) Large DB (600 MB) 15000 10000 5000 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Load (hits/sec)

  21. Adding Servers For this system, n servers meant n service queues, each operating independently, and hence less lock contention. This led to a significant increase in the workload that could be handled. However, since each server maintains its own caching mechanism, there was a distinct decrease in the cache hit probability, and an associated increase in the response time. The response time is dominated by the cache hit probability when the load is low; as the load increases the queuing for the database also increases.

  22. Multiple Servers 60000 50000 40000 1 web server avg response time (ms) 2 web servers 3 web servers 30000 4 web servers 5 web servers 6 web servers 20000 10000 0 0 1 2 3 4 5 6 7 8 Load (hits/sec)

  23. Recommendations • Projects should collect traffic data on a daily basis. • Performance measurements should be made while the software is being testing in the lab – both for new systems and when changes are being made. • Workload-based testing is a very cost effective way to do performance testing.

  24. THE END

  25. No Caching 70000 60000 50000 40000 Error Result avg resp time(ms) Form Static 30000 20000 10000 0 0 0.5 1 1.5 2 2.5 hits/sec

  26. Cache Off 30 25 20 Error Result 15 Form avg CPU cost(sec)/request Static 10 5 0 0 0.5 1 1.5 2 2.5 hits/sec

More Related