lecture

1. Software Testing Techniques

2. Software testing techniques • Software must be tested to uncover as many errors as possible through a series of test cases. • How to design test cases?

3. Software testing techniques • Software testing techniques provide systematic guidance for designing tests that • Exercise the internal logic and interfaces of every software component • Exercise the input and output domains of the program to uncover errors in program function, behavior, and performance

4. Test Case Design "Bugs lurk in corners and congregate at boundaries ..." OBJECTIVE to uncover errors CRITERIA in a complete manner CONSTRAINT with a minimum of effort and time

5. Software Testing black-box methods white-box methods Methods Strategies

6. Black-Box Testing requirements 1. Incorrect or missing function 2. Interface error 3. External data base access 4. Behavior or performance error 5. Initialization and termination errors output input

7. Black-Box Testing • Can be applied at all levels of system development • Procedures • Analyze the requirements or specifications • Determine inputs • Valid input: software under test (SUT) processes the input correctly • Invalid input: SUT detects them and handles them properly • Expected outputs for those inputs are determined • Construct the test, and execute the test • Actual outputs are compared with the expected outputs

8. Black box testing techniques • Equivalence class testing • Boundary value testing • Decision table testing • Use case testing

9. Example ages Hire Hire status

10. Example • A human resources system processes employment applications based on a person’s age. The rules are • 0-16 do not hire • 16-18 can hire on a part-time basis only • 18-75 can hire as a full-time employee • 75-99 do not hire

11. Example (cont.) If (applicantage >=0 && applicationage <16) HireStatus = “NO” If (applicantage >=16 && applicationage <18) HireStatus = “PART” If (applicantage >=18 && applicationage <75) HireStatus = “FULL” If (applicantage >=75 && applicationage <=99) HireStatus = “NO”

12. Equivalence class testing • Divides the input domain of a program into classes of data. Then identify the valid or invalid states for input conditions • The input domain is usually too large for exhaustive testing. • It is therefore partitioned into a finite number of sub-domains for the selection of test inputs. • Each sub-domain is known as an equivalence class and serves as a source of at least one test input.

13. Input domain 2 1 3 4 Example (cont.) Input domain partitioned into four classes. 15 17 37 82 Four test inputs, one selected from each class. Too many test inputs. (100 inputs) 0-99

14. Equivalence class testing • Any data value in a class is equivalent • if one test input in an equivalence class can detect a defect, all other test cases in the same equivalence class are likely to detect the same defect. • If one test input in an equivalence class does not detect a defect, no other test cases in the same equivalence class is likely to detect the defect

15. Testing invalid input? • Test 100, 106? • Based on the design • Design-by-contract -> testing-by-contract • Defensive design -> defensive testing

16. Testing-by-contract • Its approach is to create test cases only for the situations in which the pre-conditions are met. • Openfile module • File must exist • Has the name of the file • File must be openable • Has the access rights to the file….

17. Defensive testing • An approach that tests under both normal and abnormal preconditions • If the normal preconditions are met, test if the module achieve its normal postcondition • If the pre-conditions are not met, test if the modules notify the caller by returning an error code or throwing an exception correctly

18. Example - hire • Invalid inputs like -42, FRED, &*#? • Testing-by-contract • The answer is NO • Defensive testing • The answer is YES

19. Example ages Function Hire • If (applicantage >=0 && applicationage <16) • HireStatus = “NO” • If (applicantage >=16 && applicationage <18) • HireStatus = “PART” • If (applicantage >=18 && applicationage <75) • HireStatus = “FULL” • If (applicantage >=75 && applicationage <=99) • HireStatus = “NO” • if (age == 96 ) • HireStatus = “FULL” strange statement Hire status

20. How to partition? • Inputs to a program provide clues to partitioning. • Example 1: • Suppose that program P takes an input X, X being an integer. • For X<0 the program is required to perform task T1 and for X>=0 task T2.

21. How to partition?-continued • The input domain is prohibitively large because X can assume a large number of values. • However, we expect P to behave the same way for all X<0. • Similarly, we expect P to perform the same way for all values of X>=0. • We therefore partition the input domain of P into two sub-domains.

22. One test case: X=-3 Equivalence class X<0 X>=0 Equivalence class Another test case: X=15 Two sub-domains All test inputs in the X<0 sub-domain are considered equivalent. All test inputs in the X>=0 sub-domain are considered equivalent.

23. Exercise years Less than 1 year, 5 days 1 – 5 years, 10 days 5 – 10 years, 15 days, > = 10 years, 20 days vacation days

24. Example • Goofy Mortgage Company (GMC) • Income: writes mortgages incomes between $1,000/month and $80,000/month • Number of dwellings: Write a single mortgage for one through five houses • Applicant: Make mortgages only for a person • Dwelling types: Make mortgages on Condominisums, Townhouses, and Single

25. Write mortgage Income Number of dwelling Applicant Dwelling types Mortgage Qualification Qualified or unqualified

26. Example (cont.) – first input • Income: is a continuous range of value • One class of qualified values and two classes of unqualified values Qualified (valid) Three test inputs: $500 $1,500 $90,000 $80,000 $1,000 Unqualified (invalid)

27. Example (cont.) – second input • Number of dwellings: is a discrete values within a range of permissible values • One qualified class and two unqualified classes Three test inputs: 2 8 0 Qualified 0 5 1 2 3 6 4 Unqualified

28. Example (cont.) – third input • applicant: a single qualified value • One qualified and one unqualified classes Corporation Partnership … Person two test inputs: Person Corporation Qualified Unqualified

29. Example (cont.) – fourth input • Dwelling types: a set of qualified values • One qualified and one unqualified classes Farm Mobile Home …. Single Family Townhouse Condo Two test inputs: Condo Farm qualified unqualified

30. Example (cont.) - Test cases • Income: 3 • 1 qualified (valid), 2 unqualified (invalid) • Number of dwelling: 3 • 1 qualified, 2 unqualified • Applicant: 2 • 1 qualified, 1 unqualified • Dwelling types: 2 • 1 qualified, 1 unqualified

31. Example (cont.) - Test cases All inputs are qualified

32. Example (cont.) - Test cases All inputs are unqualified Which field is rejected?

33. Example (cont.) - Test cases Only one input is unqualified

34. Partitioning guidelines • Testing by contract? Defense testing? • By contract: test valid value • Defensive testing: test invalid value • Identify the equivalence classes of input or output. • By contract: at least one class – valid class • Defensive testing: • Take at least 2 classes • One class that satisfies the condition – valid class • Second class that does not satisfy the condition – invalid class • Design test cases based on the equivalence classes

35. Partitioning guidelines • Testing by contract • If an input condition specifies a range, • One valid class • If an input condition requires a specific value, • One valid class • If an input condition specifies a member of set, • One valid class • If an input condition is Boolean, • One valid class

36. Partitioning guidelines • Defensive testing • If an input condition specifies a range, • Three equivalence classes are defined • One valid class, two invalid classes • If an input condition requires a specific value, • Two equivalence classes • One valid class, one invalid class • If an input condition specifies a member of set, • Two equivalence classes are defined • One valid class, one invalid class • If an input condition is Boolean, • Two equivalence classes are defined • One valid class, one invalid class

37. Exercise • In a computer store, the computer item can have a quantity between -500 to +500. What are the equivalence classes?

38. Exercise • In a computer store, the computer item type can be P2, P3, P4 and P5 (each type influences the price ) What are the equivalence classes?

39. Exercise • Bank account can be 500 to 1000, or 0 to 4999, or 2000 (integer). What are the equivalence classes?

40. Exercise • Suppose that program P takes one character X and one string Y as inputs. • For character X: P performs task T1 for all lower case characters and T2 for upper case characters. • For string Y: P performs task T3 for the null string and T4 for all other strings. • Let us consider testing by contract

41. Exercise • We can select only 2 test inputs to cover all four equivalence classes. These are: • X: lower case, Y: null string • X: upper case, Y: not a null string • Or we can select 4 test inputs to cover all the combinations • How many should we have? • Will answer it later

42. Exercise • Payroll office uses the following rules to calculate the monthly net income (testing by contract) • Income • If the gross income is less than $3000, federal tax bracket is 15% • If the gross income is grater than or equal to $3000, and less than $5000, federal tax bracket is 20% • Otherwise, federal tax bracket is 25% • Status (F or T): • The full-time employees pay 8% to the retirement account. • The temporary employees pay 0% to the retirement account.

43. Exercise – consider invalid input • Payroll office uses the following rules to calculate the monthly salary • Income: should be greater than or equal to $500, and less than or equal to $20,000 • If gross income is less than $3000, federal tax is 15% • If gross income is between $3000 and $5000, federal tax is 20% • Otherwise, federal tax is 25% • Status: should be F or T. Display error message if user gives wrong status • The full-time employees pay 8% to the retirement account. • The temporary employees pay 0% to the retirement account.

44. Non-overlapping partitions • An equivalence class is considered covered when at least one test has been selected from it. • In the previous example, the two equivalence classes are non-overlapping. In other words the two sub-domains are disjoint.

45. Example • Suppose that program P takes three integers X, Y and Z. It is known that: • X<Y  T1, otherwise T2 • Z>Y  T3, otherwise T4

46. X<Y, Z<=Y X=4, Y=7, Z=1 X>=Y, Z<=Y X=4, Y=2, Z=1 X<Y X<Y, Z>Y X=1, Y=7, Z=9 X>=Y Z>Y Z<=Y X>=Y, Z>Y X=4, Y=2, Z=9 Example (cont.)

47. Example (cont.) • In this example, we could select 4 test cases as: • X=4, Y=7, Z=1 satisfies X<Y • X=4, Y=2, Z=9 satisfies X>=Y • X=1, Y=7, Z=9 satisfies Z>Y • X=1, Y=7, Z=2 satisfies Z<=Y • Thus, we have one test case from each equivalence class combination.

48. Example (cont.) • Can we reduce it to 2 test inputs and satisfy all four equivalence classes? • X=4, Y=7, Z=1 satisfies X<Y and Z<=Y • X=4, Y=2, Z=9 satisfies X>=Y and Z>Y

49. Weak Normal Equivalence testing • Assumes the ‘single fault’ or “independence of input variables.” • e.g. If there are 2 input variables, these input variables are independent of each other. • Partition the test cases of each input variable separately into different equivalent classes. • Choose the test case from each of the equivalence classes for each input variable independently of the other input variable

50. Example of : Weak Normal Equivalence testing Assume the equivalence partitioning of input X is: 1 to 10; 11 to 20, 21 to 30 and the equivalence partitioning of input Y is: 1 to 5; 6 to 10; 11 to 15; and 16 to 20 We have covered everyone of the 3 equivalence classes for input X. X 30 For ( x, y ) we have: (24, 2) (15, 8 ) ( 4, 13) (23, 17) 20 10 1 Y 1 5 10 15 20 General rule for # of test cases? What do you think? # of partitions of the largest set? We have covered each of the 4 equivalence classes for input Y.