System Testing Principles and Techniques

1. Chapter 9 Testing the System

2. Learning objectives Learning Objectives: After studying this chapter, you should be able to: • Describe how system testing differs from unit and integration testing. • Classify tests as function testing, performance(性能) testing, acceptance testing or installation testing. • Understand the purposes and roles of function testing, performance testing, acceptance testing, and installation testing. • Define software reliability(可靠性), maintainability and availability(可用性，有效性). • Describe different techniques for measuring reliability, maintainability and availability. • List the different types of test documentation and know what items belong in test documentation. • Understand the special problems associated(相关的,结合的) with testing safety-critical(临界的,决定性的) systems. • Describe the principles(原理) of Cleanroom(净室) and how it differs from conventional(常规的) testing.

3. 9.1 PRINCIPLES OF SYSTEM TESTING • Sources(来源) of Software Faults FIGURE 9.1 Causes of faults during development

4. 9.1 PRINOPLES OF SYSTEM TESTING System Testing Process There are several steps in testing a system: • 1. function testing • 2. performance testing • 3. acceptance testing • 4. installation testing

5. 9.1 PRINOPLES OF SYSTEM TESTING • Process objectives. • Build or Integration Plan. • Configuration Management(配置管理) • Versions(文本,版本,方案,译本) and Releases(版本,发布). • Regression Testing(回归测试). 1. Inserting your new code 2. Testing functions known to be affected by the new code 3. Testing essential(基本,本质) functions of m to verify that they still work properly (the actual regression testing) 4. Continuing function testing of m + 1

6. 9.1 PRINOPLES OF SYSTEM TESTING • Deltas(δ,增量), Separate Files(分开的,单独的), and Conditional Compilation(条件编译) • Change Control(更改控制). • Test Team • Professional testers organize and run the tests. • System designers

7. 9.2 FUNCTION TESTING • Purpose and Roles(目标与任务) • have a high probability of detecting a fault(尽可能地发现错误) • use a test team independent of the designers and programmers • know the expected actions and output • test both valid and invalid input(有效与无效输入都测试) • never modify the system just to make testing easier(决不因简化测试而修改系统) • have stopping criteria(有停止测试的标准) • Function testing is performed in a carefully controlled situation.

8. 9.2 FUNCTION TESTING • Cause-and-Effect(因果关系) Graphs Cause-and-effect graph.

9. 9.3 PERFORMANCE TESTING(性能测试) • Purpose and Roles(目的与任务) Performance testing examines how well the calculation(计算) is done; the speed of response(响应) to user commands, accuracy(准确性) of the result, and accessibility(可接近性；可达性) of the data are checked against the customer's performance prescriptions(规定,指示)

10. 9.3 PERFORMANCE TESTING(性能测试) • ◆Types of Performance Tests • Stress tests(强度测试). • Volume tests(批量测试). • Configuration tests(配置测试). • Compatibility tests(兼容性). • Regression tests(回归测试). • Security tests(安全性测试). • Timing tests(实时测试). • Environmental tests(环境试验). • Quality tests(质量测试). • Recovery tests(修复测试). • Maintenance tests. • Documentation tests. • Human factors tests(人为因素测试)

11. 9.4 RELIABILITY(可靠性), AVAILABILITY(有效性) AND MAINTAINABILITY(可维护性) • Definitions(定义) • Reliability involves behavior over a period of time, but availability describes something at a given point in time. • software availability is the probability(可能性) that a system is operating successfully according to specification at a given point in time. • software maintainability is the probability that, for a given condition of use, a maintenance activity can be carried out within a stated time interval and using stated procedures and resources.

12. 9.4 RELIABILITY(可靠性), AVAILABILITY(有效性) AND MAINTAINABILITY(可维护性) • Measuring Reliability Availability' and Maintainability • Mean Time to Failure (MTTF). (失效时间, 故障时间,无故障工作时间) • Mean Time to Repair (MTTR)( 平均修复时间) Mean Time between Failures (MTBF)( 平均故障间隔时间), MTBF = MTTF + MTTR reliability as R = MTTF/(1 + MTTF) A = MBF/(1 + MTBF) M = 1/(1 + MTTR) • Reliability(可靠性) Stability(稳定性) and Growth(扩充,生长)

13. 9.4 RELIABILITY(可靠性), AVAILABILITY(有效性) AND MAINTAINABILITY(可维护性) • Reliability Prediction(可靠性预测) • Importance(重要性) of the Operational Environment statistical testing(统计测试) 1. Testing concentrates on the parts of the system most likely to be used and hence should result in a system that the user finds more reliable. 2. Reliability predictions based on the test results should give us an accurate prediction of reliability as seen by the user. However, it is not easy to do statistical testing properly There is no simple or repeatable way of defining operational profiles. We see later in this chapter how clean-room software development integrates statistical testing into its approach to building quality software.

14. 9.5 ACCEPTANCE TESTING • Purpose and Roles(目标与任务) The purpose of acceptance testing is to enable the customers and users to determine if the system we built really meets their needs and expectations. Thus, acceptance tests are written, conducted(实施的), and evaluated by the customers, with assistance(协助) from the developers only when the customer requests an answer to a technical question.

15. 9.5 ACCEPTANCE TESTING • Types of Acceptance Tests • A pilot test(小规模试验) alpha test(α测试)，beta test(β测试)，parallel testing(并行测试). • Results of Acceptance Tests

16. 9.6 INSTALLATION TESTING • The final round of testing involves installing the system at user sites. • Installation tests require us to work with the customer to determine what tests are needed on-site. When the customer is satisfied with the results, testing is complete and the system is formally delivered.

17. 9.7 AUTOMATED SYSTEM TESTING • simulator(模拟器；模拟程序；模拟系统)

18. 9.8 TEST DOCUMENTATION • Test Plans • guide the management of testing • guide the technical effort required during testing • establish(建立,制定) test planning and scheduling, including specifying equipment needed, organizational requirements, test methods, anticipated outcomes, and user orientation • explain the nature and extent(程度，范围，区域) of each test • explain how the tests will completely evaluate system function and performance • document test input, specific test procedures, and expected outcomes

19. 9.8 TEST DOCUMENTATION • the test schedule includes 1. the overall testing period 2. the major subdivisions(子部分,子划分) of testing, and their start and stop times 3. any pretest(预先试验) requirements (such as orientation(定向,定位) or familiarization(熟悉,通晓) with the system, user training, or generation of test data) and the time necessary for each 4. the time necessary for preparing and reviewing the test report

20. 9.8 TEST DOCUMENTATION • Test Specification(测试技术要求,测试规范) and Evaluation(评价) • Is the system using actual input from users or devices, or are special cases generated by a program or surrogate(代用) device? • What are the test coverage criteria(覆盖标准)? • How will data be recorded? • Are there timing, interface, equipment, personnel, database, or other limitations(局限性) on testing? • If the test is a series of smaller tests, in what order are the tests to be performed?

21. 9.8 TEST DOCUMENTATION • Test Analysis Report(测试分析报告) • It documents the results of a test. • If a failure occurs, the report provides information needed to duplicate(复制,转录) the failure (if necessary) and to locate and fix(固定) the source of the problem. • It provides information necessary to determine if the development project is complete. • It establishes(建立) confidence(信心) in the system's performance.

22. 9.8 TEST DOCUMENTATION • Problem Report(问题报告) Forms(格式,表格) • Location(定位,位置): Where did the problem occur? • Timing: When did it occur? • Symptom(症状): What was observed? • End result: What were the consequences? • Mechanism(机制): How did it occur? • Cause: Why did it occur? • Severity(严重性): How much was the user or business affected? • Cost: How much did it cost?

23. 9.9 TESTING SAFETY-CRITICAL(临界安全) SYSTEMS