Chapter 20 Verification and Validation Testing

2. Obviously, it doesn�t do much good to write lots and lots of software If it doesn�t work, or If it can�t be trusted,or If it can�t be easily modified and maintained. Edward Yourdon [yourdon92]

4. Introduction Testing is performed to validate Features and /or verify design. The cost for testing activity runs from 25% to 40% of the entire process [Boehm79]. The keys to successful testing are: well-designed and timely test planning and strategy, development of effective test cases, and efficient and effective management to test activities.

6. Test Planning and Strategy White-Box Testing VS. Black-Box Testing Testing is an activity to find Defects in a program by executing the program. Two things drive a testing activity: logic (white box testing) and data (black box testing).

7. While-Box Testing To execute white-box testing, the tester must know the internal structure of the program. You can completely test every statement of the program by executing each statement at least once. The control flow through a program may contain an infinitely large number of unique logic paths.

8. Black-Box Testing The tester views the program as a black box in which internal operations cannot be seen. The tester makes sure that a program produces expected output and behaves to meet each Feature specification.

9. Orderly Testing Process

11. Unit Test Unit test activity tests the individual Chunks which are composed of objects, functions, procedures, subroutines, or any small building blocks of a Software Product. The unit test should be performed along with the development of each unit. Unit testing is generally white-box oriented. The idea is to test each building component of a program before combining them into a whole

12. The benefit unit testing include: Unit testing leads to an overall efficient software life cycle as multiple units can be testes simultaneously and the testing activities can proceed in parallel with development activities. Since unit testing is focused on functionality and internal behavior, finding and correcting Defects is easier. Building an entire system with Defect free units contributes to higher quality software in less time.

13. System Test System testing (sometimes called integration testing) begins after the integration of all tested units. The objective of system testing is to determine if the integrated units work together as designed. As a test is normally required to assure that a system is properly integrated and is ready for the more comprehensive � testing process.

14. Feature Test The objective of a Feature test is to validate the Software Product operates correctly according to proposed Features. Feature test is, in general, black-box oriented. The testers do not care how the program behave internally, but they check to see if the program behaves according to the required Features. What test engineers do at this stage is to profile general users. Developers should normally perform unit and system testing. Feature testing, however, should be performed by people other than developers In order to perform an effective Feature test, a complete and detailed Feature specification must be available.

15. Qualification Test Qualification test determine if the SoftwareProduct is acceptable for release to the market. Test frequently used to qualify SoftwareProducts are: Durability Stress Performance Usability Compatibility

16. Durability During durability testing, the system is heavily exercised. Examples: If an operating system of firmware is being tested, the system may be continuously run for several nights. If the program is a database system, large amounts of data may be entered to see if the system is durable.

17. Stress A stress test tries to hit the broader line of system capability. Executing a program with just above the required memory size is an example of a stress test.

18. Performance Performance frequently becomes a tradeoff with resources (or cost). A specified level of performance has to be obtained to release a SoftwareProduct.

19. Usability Usability Testing determines the existence of usability Defects. A Usability Defect is anything in the SoftwareProduct which prevents a target user completing a target a task with reasonable effort and within reasonable time.

20. Compatibility Compatibility testing determines if software is compatible with a company�s previous SoftwareProducts or similar SoftwareProducts of targeted competitors.

21. Regression Test The objective of regression testing is to determine if the local change in the program has caused Defects in other aspects of the program. By running an entire test suite, we can achieve the objective if time and cost are not any issue. However, reality is that we always need to evaluate cost/benefit and time constraint very carefully. Thus, usually a subject of test cases is run to exercise the changed area and the area that could be affected by the changes.

22. Installation Test Installation test is the last validation step for a commercial SoftwareProduct. To distribute a SoftwareProduct, you must select some type of secondary storage for the software such as display units, disks, CD�s or Tapes.

23. Software Testing Integrated Environment There are many tools being developed to assist in the conduct and management of testing activities. Current trends of the Computer Aided Software Testing (CAST) tool market are in the direction of an integrated test environment. Test Case Development. Suitable test cases are necessary for a successful test process. Effective testing has two parts: Test execution and Test configuration including required test data.

24. Test Case Development Tools Since test execution is repeatedly performed, your time and effort required to develop test cases will be well spent. Many tools are commercially available to assist in developing test cases.

25. Test Design Tools Test design tools help you organize test plans and develop test cases from the set of software Features and design specifications.

26. Profiling Tools Profiling tools allow tester (or developers) to understand the behavior and objective of a program. A memory profiling tool has been developed for use within Hewlett-Packard. Understanding how a program is being executed and monitoring internal resource usage helps crate effective test cases.

27. Code Coverage Tool A code coverage tool provides information regarding how much of the code has been executed (or activated) by the test cases running against the code. Generally speaking, coverage tools are used for a white-box oriented environment. They monitor the program during execution, providing statistics on code statement and decision statement coverage. One of the drawbacks of using code overage tools is overhead. Coverage tools require instrumentation, slow down test execution and impose a heavy overhead.

28. Test Data Management Tool Test cases may be reused for several SoftwareProducts. A rest data management tool should have access to a database of test cases. The test database is useful for managing test activity, selecting test cases (particularly for regression testing), and enhancing test cases for future test activities.

29. Test Execution Tools Test execution tools are for test execution activity and for comparing results after execution.

30. Capture and Play-Back Tool A capture play-back tool is useful if you are developing window based software. The idea is that a person runs a test case from a terminal. The tool records keystrokes or mouse movements that the test subject performs. Once a complete test sequence has been performed, the recorded movements can be replayed.

31. Test Driver Tools Test driver tools apply inputs to SoftwareProducts to stimulate operations.

32. Data Compare Tools A data compare tool checks data bit-by-bit or character-by character. When a capture and play-back tool is used, the output can be saved in a file. The data compare tool then compares the file bit-by-bit or character-by-character against output form a test.

33. Data Generator Tool A data generator tool produces random streams of input data for a program under test. It has been used in formulating data to test database or financial systems software. When a system takes a variety of input formats and the system has to be tested with many different formats of data, a data generator is useful

34. Managing Testing Activities and Results Tools that facilitate managing test activity: Test Activity Monitor Tools � Commercially available databases system have Features that can prompt tester or manager if a test activity is not as scheduled. Intelligent agents to alert managers can be developed for the PAMPA tool described in Chapter 15. In this section, we will discuss Defect tracking systems. Defect Tracking System � Defect tracking systems are used by many Organization. Some are commercial off-the shelf (COTS) and other are in-house developed tools.