CS 47 23 Software Validation and Quality Assurance

1. CS4723Software Validation and Quality Assurance Lecture 02 Overview of Software Testing

2. Approach to remove bugs Testing Feed input to software and run it to see whether its behavior is as expected Limitations Impossible to cover all cases Test oracles (what is expected) Static checking Identify specific problems (e.g., memory leak) in the software by scanning the code or all possible paths Limitations Limited problem types False positives 2

3. Approach to remove bugs Formal Proof Formally prove that the program implements the specification Limitations Difficult to have a formal specification The proof cost a lot of human efforts Inspection Manually review the code to detect faults Limitations: Hard to evaluate Sometime hard to get progress 3

4. Answer is testing, why? “50% of my employees are testers, and the rest spends 50% of their time testing” ---- Bill Gates, 1995 More reliable than inspection, relatively cheap Actually in the old days, when testing is expensive, inspection was the major answer You get what you pay (linear rewards) Compared to other 3 approaches Inspection, static checking, formal proof 4

5. Testing: Concepts Test case Test oracle Test suite Test script Test driver Test coverage 5

6. Testing: Concepts Test case An execution of the software with a given list of input values Include: Input values, sometimes fed in different steps Expected outputs Test oracle The expected outputs of software by feeding in a list of input values A part of test cases Hardest problem in auto-testing: test oracle problem 6

7. Testing: Concepts: Example 7

8. Testing: Concepts Test suite A collection of test cases Usually these test cases share similar pre-conditions and configuration Usually can be run together in sequence Different test suites for different purposes Smoke test, Certain platforms, Certain feature, performance, … Test Script A script to run a sequence of test cases or a test suite automatically 8

9. Testing: Concepts Test Driver A software framework that can load a collection of test cases or a test suite It can usually handle the configuration and comparison between expected outputs and actual outputs Test Coverage A measurement to evaluate how well the testing is done The measure can be based on multiple elements Code Input combinations Specifications 9

10. Granularity of Testing: V-model 10

11. Granularity of testing Unit / Module Testing Test of a single module Integration Testing Test the interaction between modules System Testing Test the system as a whole, by developers on test cases Acceptance Testing Validate the system against user requirements, by customers with no formal test cases 11

12. Stage of software Testing Development-time testing Unit testing, Integration Testing Before-release testing System testing, Acceptance Testing User testing Actual usage -> field bugs & patches 12

13. Types of testing by how they are designed Black box testing The tester are just like normal users They just try to cover input space and corner cases White box testing The tester knows everything about the implementation They knows where the bugs are more probably be They can exercise paths in the code 13

14. Black Box Testing: General Guidelines Divide value range and cover each part Cover boundary values Try to reach all error messages Try to trigger potential exceptions Feed invalid inputs wrong formats, too long, too short, empty, … Try combinations of all above Repeat same and use different inputs for many times if the input is a sequence 14

15. Black Box Testing Techniques Boundary value testing Boundary value analysis Robustness testing Worst case testing Equivalence class testing Decision table based testing 15

16. Boundary Value Analysis Errors tend to occur near the extreme values of an input variables Boundary value analysis focuses on the boundary of the input space to identity test cases Boundary value analysis selects input variable values at their Minimum Just above the minimum A nominal value Just below the maximum Maximum 16

17. Example of Boundary Value Analysis Assume a program accepting two inputs y1 and y2, such that a < y1< b and c < y2 < d . . .. ... . . 17

18. Single Fault Assumption for Boundary Value Analysis Boundary value analysis is also augmented by the single fault assumption principle “Failures occur rarely as the result of the simultaneous occurrence of two (or more) faults” In this respect, boundary value analysis test cases can be obtained by holding the values of all but one variable at their nominal values, and letting that variable assume its extreme values 18

19. Application Scenario of Boundary Value Analysis Several independent variables that represent bounded physical quantities No consideration of the function of the program, nor of the semantic meaning of the variables Good news: We can distinguish between physical and other types of variables 19

20. Generalization of Boundary Value Analysis The basic boundary value analysis can be generalized in two ways: By the number of variables - (4n +1) test cases for n variables By the kinds of ranges of variables : map to order Strings Sequences Complex Data Structures, e. g., trees 20

21. Robustness Testing A simple extension of boundary value analysis In addition to the five boundary value analysis values of variables, we add values slightly greater that the maximum (max+) and a value slightly less than the minimum (min-) The main value of robustness testing is to force attention on exception handling 21

22. Example of Robustness Testing . . . ... .… . . . 22 UTSA CS3773

23. Worst Case Testing No single fault assumption: error happens when more than one variable has an extreme value Considering n inputs in boundary analysis, we take the Cartesian product of the five values for 1, 2, 3, … n variables We can have 5n test cases for n input variables The more interactions on inputs -> more on worse case testing Input partitions: Length & Width vs. Length & price 23

24. Example of Worst Case Testing .. ... .. ... .. ... .. ... .. ... 24

25. Equivalence Class Testing Divide the value range of an input to a number of subsets Subsets are disjoint The union of the subset if the value range Values in one subset does not make difference for the software concerned Water temp in a car: <32, 32 – 212, >=212 Normal colors vs. Metallic colors 25

26. Example of Equivalence Class Testing 26

27. Equivalence Class Testing The use of equivalence class testing has two motivations: Sense of complete testing The entire set is represented provides a form of completeness Avoid redundancy The disjointness assures a form of non-redundancy Note Also check boundaries Combinations of inputs also follows the rule: more interaction -> more combinations 27

28. Equivalent Class for non-numeric inputs Feature extraction For string and structure inputs Split the possible value set with a certain feature Example: String passwd => {contains space}, {no space} It is possible to extract multiple features from one input Example: String name => {capitalized first letter}, {not} => {contains space}, {not} => {length >10}, {2-10}, {1}, {0} One test case may cover multiple features 28

29. Equivalent Class for non-numeric inputs Feature extraction For string and structure inputs Split the possible value set with a certain feature Example: String passwd => {contains space}, {no space} It is possible to extract multiple features from one input Example: String name => {capitalized first letter}, {not} => {contains space}, {not} => {length >10}, {2-10}, {1}, {0} One test case may cover multiple features 29

30. Equivalent Class for non-numeric inputs Feature extraction: structure input A Word Binary Tree (Data at all nodes are strings) Depth : integer -> partition {0, 1, 1+} Number of leaves : integer -> partition {0, 1, <10, 10+} Root: null / not A node with only left child / not A node with only right child / not Null value data on any node / not Root value: string -> further feature extraction Value on the left most leaf: string -> further feature extraction … 30

31. Equivalent Class for non-numeric inputs Infeasible feature combination? Example: String name => {capitalized first letter}, {not} => {contains space}, {not} => {length >10}, {2-10}, {1}, {0} Length = 0 ^ contains space Length = 0 ^ capitalized first letter Length = 1 ^ contains space ^ capitalized first letter 31

32. Decision Table Make it easy to observe that all possible conditions are accounted for Decision tables can be used for: Specifying complex program logic Generating test cases with oracles 32

33. Example of Decision Table Printer Troubleshooting 33

34. Decision Table Usage The use of the decision-table model is applicable when: Specification is given or can be converted to a decision table The order in which the predicates are evaluated does not affect the interpretation of resulting action Note: Decision table needs not cover all combinations 34

35. White Box Testing: General Guidelines Try to cover all branches Study the relationship between input value and branch logic Test more on complex modules Measure complexities of modules by code size, number of branches and loops, number of calls and recursions 35

36. White Box Testing: Techniques More difficult than black box testing Need to understand the code Code Coverage Guided Automatic support Symbolic execution Complexity measurement and Defect prediction 36

37. Review: Test overview Test is the practical choice: the best affordable approach Concepts: test case, test oracle, test suite, test driver, test script, test coverage Granularity: unit, integration, system, acceptance Type by design principle: black-box, white-box Black-box-testing: boundary, equivalence classes White-box-testing: branch coverage, complexity 37