Performance Assessment for Web-based Systems

1. Performance Assessment for Web-based Systems Min Song, Ph.D. IS 465

2. Topology of a Web Site Internet I S P F i r e w a l l Browser Security Server Web Server Browser Database Server Browser Application Server

3. General Considerations • Application conception • What’s to be done? • What’s the whole purpose of this task? • Design • How is it going to be done? • Flexibility, Dynamic • Development • What tools to use? • What technologies to implement? • TESTING • What? it’s working well, isn’t it? • Deployment • How soon?, Pronto!, Is fast isn’t it?

4. The NEED to Test(Are we wondering…) • Is the site content meaningful? • Is this application easy to use? • How about browser compatible? • Are we using the right tools? • How reliable is our technology? • Do the Servers have enough power? • How many visitors are we expecting? • Are the machines fast enough? • How much activity can the site handle?

5. What and How to test?(Fundamental Considerations) • Test Planning • Browser Compatibility • Design compliance • Usability • Efficiency • Scalability • Best suited OS platform • Network bandwidth • Hardware capacity • Site availability • Monitoring • Security

6. Testing a Web Site Understanding what types of testing are needed for each application, is essential for several reasons: • Ensures you that the testing effort, will support the business and technical requirements. • The activities required for each type of test and its associated phases, are included within the master test plan (and schedule). • Helps identifying and plan for the environments and resources that are needed to prepare and execute each test type. • Assures the achievement of any Testing Goals.

7. Testing a Web Site (continued) There are mainly two essential types of testing that should be accomplished. • Functionality • Performance

8. Functionality Testing • What is it? • Why is so important? • What is: • Unit Test • Integrated Test • Functional Test • System Test • Regression Test

9. What is Functionality Testing? • Is a set of skilled approaches and procedures which systematically verify that the application’s components and process modules, are available and working as expected. Compliance with Business rules, Technical requirements, System Documentation, and User manuals is of course… Mandatory!

10. Why is Functionality Testing important? Missing Image

11. Unit Test Unit tests perform basic tests at component level and test a specific business process, application, and/or system configuration. Unit tests ensure that each unique path of a business process performs accurately to the documented specifications and contains clearly defined inputs and expected results Functionality Testing -Unit Test

12. Integration Test Integration test is testing a specific feature together with other newly developed features Testing the interface of two components explores how components interact with each other Integration testing inspects the variables passed not only between two components, but also the global variables. Integration testing is specifically aimed at exposing the problems that arise from the combination of components Previously captured unit test scripts can be combined to create a variety of integration test cases This test phase assumes the components and the objects they manipulate have all passed their local unit tests Functionality Testing -Integration Test

13. Functional Test Functional testing is centered around the following items: Valid Input - identified classes of valid input must be accepted. Invalid Input - identified classes of invalid input must be rejected Functions - identified functions must be exercised Output - identified classes of application outputs must be exercised. Systems/Procedures - interfacing systems or procedures must be invoked. Functionality Testing - Functional Test

14. Regression Test Regression tests are a series of tests to ensure that no adverse changes are introduced to the application during maintenance changes, upgrades, or other modifications. Regression testing is composed of a suite of components that are rerun when any one-application component has been modified This provides confidence that the system will function with newly implemented requirements. Regression testing is often an iterative process during a testing cycle Functionality Testing – System & Regression Test System Test • System testing studies issues and behaviors that can only be exposed by testing the entire integrated system • Verifying these characteristics is very important for products intended for users on diverse OS and Hardware platforms. • Identification of faulty Hardware and Software components, is feasible this way.

15. Performance Testing • What is it? • Why is so important? • What are the types of Performance Tests? • What is the difference between Load, Performance, and Stress Testing ?

16. What is Performance Testing ? • Performance Tests determine the runtime “behavior” of the application and its supporting infrastructure, under certain conditions. • Performance Testingis used to measure several system characteristics, such as processing speed, response time, resource consumption, throughput, and efficiency.

17. Why is Performance Testing important? • The week of Feb 6, 2000: Hackers delivered over 1-Billion transactions concurrently to each one of these sites. • Yahoo • eBay • Buy.com • Amazon • eTrade How did you think they performed ?

18. Load Test Load Tests determine the applications behavior under load, up to and including its limits (not just at its limits). Load tests specifically refer to the load size (number of concurrent users) and related values. Performance Testing - Load Test

19. Stress Test Stress testing determines the applications ability to handle large amounts of data. Stress Testing can be much more successful with a full load applied to the server. Running addition applications on the server in conjunction with the client side tests is an additional form of stress testing. Performance Testing - Stress Test

20. Performance, Load, and Stress Testing ? They all have different meanings, although they seem to be used almost interchangeably sometimes. To better understand these differences; let’s apply these terms while testing a car: • Performance Testing - a car does 0-60mph in a certain number of seconds under normal conditions • Load Testing – The car can pull a trailer weighing x - tons before it can no longer move • Stress Testing - The car continues to work fine driving 60 mph for two hours, in 120 degree temperatures, with only 1 gallon of coolant in the radiator

21. Why Automate Testing? • Is it necessary to repeat the sequence of actions many times? • Do you need to run the same tests on multiple hardware configurations? • Need to test with several concurrent users? • Exercise multiple options, components, and configurations • Save time and money

22. What are we testing for? • Where should I start? • How do we proceed? • What tool do I select?

23. Simulating the Real World • The MAIN purpose of running automated tests is to emulate the REAL world as close as possible while saving time and effort.

24. Tips for selecting an Automated Testing Tool • Ease to use (and learn) • Standards compliance • Robust and Reliable (accurate) • Flexible • Affordable • Maintenance and Support • Important: Is it the right tool for the job?

25. Everyone involved should be clear on what exactly we are testing for and the goals for the outcome of the test Testing For: How many Total Concurrent Users ? At what point does the performance degrade ? Which server will max out first ? (Web, DB, App, etc.) How about Data integrity ? What is the Total Throughput ? (Bandwidth, pipeline) Goals: Round time =< 15 sec Total Concurrent Users => 300 / Min What are we Testing for?

26. 1.Define 2.Record 3.Modify 4.Execute 5.Analyze Automation Web Testing Methodology 5-Phases Modeling Paradigm

27. The Application Describe the application to test using a application model Describe the Hardware environment Create a Benchmark (Agenda) to be recorded in Phase 2. Define what tasks each user will perform Define (or estimate) the percentage of users per task. Benchmark - A benchmark procedure uses a predefined set of data and measures the results returned by the Application Being Tested (ABT). Benchmarks are intended to make comparisons easy! 1.Define

28. Testing Types and Metrics needed Load Increasing Workload Start / End with how many users? Increment by how many users? How often to increment (Seconds)? Steady-State How many users? How long (Days/Hours/Minutes/Seconds)? Performance What are the reasonable response times in seconds? (each task/page) Stress What are the expected limits? How many Peak users ? 1.Define

29. Record Record the defined testing activities that will be used as a foundation for your load test scripts. One activity per task or multiple activities depending on user task definition 1.Define 2.Record 3.Modify • Modify • Modify load test scripts defined by recorder to reflect more realistic Load test simulations. • Defining the project, users • Randomize parameters (Data, times, environment) • Randomize user activities that occur during the load test

30. Test Date: 07/17/2001 Virtual Users (VUs):Test Goals Start: 5 Max Cycle Time <= 40 Sec Incremented by: 5 Maximum: 200 Initial Cycle Time = 35 Sec Sleep = 32 Seconds Test Script(s): One typical user from logon through completion (continuously) This would be an average user Performance Test scenario 4. Execute

31. 5. Analyze Performance Test scenario Cycle Time starts to oscillate, indicating a bottleneck

32. Performance Test scenario (Continued) The CPU Started making large spikes right from the start

33. Performance Test scenario (Continued) The TPS & Throughput peaked out indicating another Bottleneck

34. Performance Test scenario Both Load Generator CPU’s are spiking to 100%

35. Conclusions • Define the purpose of taking on a test automation effort • Develop a test automation strategy (map out what's to be automated) • Treat the automated process as any other software development task • Do your homework (Investigate!) • Test early and test often! • Remember: • The effort of test automation is an investment • Those performing the work should have the correct skill sets • YOU SHOULD TEST UNDER LOAD!!!