Testing

2. Testing • Testing is a critical process within any software development and maintenance life cycle model. • Testing is becoming increasingly important to the software development industry because of its implications on the quality of the produced software and the reputation of the producing company. • Testing activities include testing specifications, designs and implementations. • Testing implementations is often referred to as software testing, or simply testing.

3. Test coverage • Only exhaustive testing can guarantee full coverage. • Exhaustive testing is impractical since it may take years to complete even for testing small size programs. • Consider testing the software when a first name text field of 20 characters maximum is filled by a user. Exhaustive testing requires that we test the software for all possible correct and incorrect field input values. • Assume 80 possible correct and incorrect characters, the number of possible combinations for a valid or invalid first name would exceed 8020. Using a computer that takes, say, 50 nanoseconds to process one combination, we would need about 1011 years to complete the testing for all possible combinations. It is indeed impractical!

4. The reality • With the absence of exhaustive testing, there are no formal guarantees that soft­ware is error-free. • In fact, it may happen in practice that even when all errors have been detected during testing, they may not all be fixed due to the lack of time. • Unfixed errors may be fixed in future releases of the software. • The difficulty in software testing is the inability to quantify the errors existing in software, and hence predict how many errors are left undiscovered.

5. Test coverage experiment • Random testing experiment: errors of different levels of complexity were inserted in the software under test • All errors except for one were discovered progressively. • No indication on how much more testing is needed • Finding errors is an asymptotic nonlinear function of time.

6. Dynamic and static testing • Dynamic testing • Test the software by executing it and applying test inputs • Observe the output and make a judgment to take a verdict • Static testing • Check code without executing it • Automatic static testing: Code as an input to a static analysis tool • Non-automatic static testing: review, walkthrough, inspection

7. Test case • Test plan includes a test suite • A test suite is a set of organized test cases • Organize around use cases for example • Template to describe a test case • Id, name, reference, priority, preconditions, preamble, input, expected output, observed output, verdict and remarks • Test cases are obtained using known testing techniques and strategies

8. Test case design vs execution • Test case is developed first – verdict and remarks are empty • Test case is executed when the actual testing is performed – the verdict is obtained • Typically test cases are developed (designed) before even writing any code • Acceptance test cases in the acceptance test plan (ATP) can be developed after requirements specification phase • Unit and integration test plans can be developed after the design is completed and prior to coding

11. Test Case (TC) Example TC#:S-221 TC Description:This test is to check the response to an invalid input selection. TC Precondition:Go to screen Flight_Management TC Input:Enter <F7> TC Expected Output:Error message: “Invalid input, Enter “1” to “4” or ESC” OOP Development

12. White-box, black-box and grey-box testing techniques • White-box: module or unit testing • Black-box: module testing and system testing

13. Black-box unit/module testing techniques • Boundary value analysis • Equivalence class testing • Decision table / decision tree based testing • State based testing • Robustness testing

14. Boundary value analysis • BVA: test inputs obtained from the boundaries • Serious errors hide at the boundaries. • For example, simple boundary values for an input of integer data type would include -INT_MAX, -INT_MAX + 1, -1, 0, +1, INT_MAX - 1 and INT_MAX. • For complex and composite data types, boundary values become more difficult to generate and may require more ingenuity to obtain them. • Moreover, if interface specification includes more than one input, different test cases considering possible feasible combinations of BVs for each input must be considered

15. BVA Example • Suppose you are given an executable code including a function called digitCount whose functional interface specification states that digitCount takes an integer n as an input parameter, and returns an integer representing the number of digits in n. • Test cases whose test inputs are selected based on boundary value analysis are partially shown in Table 8.3. • Assume that for a 32-bit integer, INT_MAX is +2147483647.

16. Testing function digitCount based on boundary value analysis

17. Example 8.2 – Testing search function

19. Equivalence class testing • Partitioning input domain into disjoint subsets or partitions • Partition includes related values • Consider one test input from each partition • Equivalence relation among partition elements • ‘having the same number of digits’ • ‘all positive integers’ • ‘all even numbers’ • ‘all prime numbers’ • ‘all powe or 2 numbers’ • …..

21. Decision table based testing • Decision table: each column denoting a rule becomes the basis of a test case • Conditions produce inputs • Actions produce expected outputs • In table 4.1: input variables are age, history and speed, and output variables are penalty and points

23. Decision tree-based testing • Each distinct path from the root node to a leaf node is the basis of a test case • If along the path, a variable input value does not exist, the test input for that variable can be assigned any valid value. • For example, in fig 4.23, for the test case corresponding to the leftmost path, the speed value can be any illegal speed

25. State-based testing • An implemented finite state machine conforms to its specification, if the observed output sequence matches the expected output according to the specification • Test sequence must cover all states transitions • Used to test do black box testing of systems • GUI, protocols, web page connectivity, … • Transition tour: a sequence covering all transitions and returning back to the initial state • Can only cover output errors

27. Black box-based robustness testing • Test a system using one or preferably more black box based testing method • Test for incorrect and illegal inputs also • In equivalence class testing, select one representative bad input from every partition of the bad input values • For FSM based systems, apply unexpected inputs at each state – the system should survive • For boundary value testing, test for illegal values beyond the boundaries – the system should survive

28. White-box testing techniques • Given the source code of the module to test • Control flow based testing techniques • Data flow based testing techniques • Path testing

29. Control flow based testing • Given source code of a module, transform it to a control flowgraph • Nodes and edges – decision nodes, joining nodes

33. Example: digitCount 1 int digitCount(int n) 2 { 3 int count = 0; 4 while (n != 0) 5 { n = n / 10; 6 count++; 7 } 8 return count; 9 }

34. White-box testing • Statement coverage • Graph based • Branch coverage • Condition coverage • Path coverage OOP Development

35. Example: Statement Coverage 1 int maxPositive(int n1, int n2) { 2 if (n1<=1 && n2<=1) 3 n1=1; 4 if (n1<=n2) 5 n1=n2; 6 return n1; } TC Inputs: n1=n2=1 OOP Development

36. 1,2 3 4 5 6 TC#1 Inputs: n1=n2=1 TC#2 Inputs: n1=2, n2=0 Example: Branch Coverage 1 int maxPositive(int n1, int n2) { 2 if (n1<=1 && n2<=1) 3 n1=1; 4 if (n1<=n2) 5 n1=n2; 6 return n1; } OOP Development

37. TC#3 Inputs: n1=0, n2=2 Example: Condition Coverage 1 int maxPositive(int n1, int n2) { 2 if (n1<=1 && n2<=1) 3 n1=1; 4 if (n1<=n2) 5 n1=n2; 6 return n1; } 1,2 3 4 5 TC#1 Inputs: n1=n2=1 6 TC#2 Inputs: n1=2, n2=0 OOP Development

38. TC#4 Inputs: n1=n2=0 Example: Path Coverage 1 int maxPositive(int n1, int n2) { 2 if (n1<=1 && n2<=1) 3 n1=1; 4 if (n1<=n2) 5 n1=n2; 6 return n1; } 1,2 3 4 5 TC#1 Inputs: n1=n2=1 6 TC#2 Inputs: n1=2, n2=0 TC#3 Inputs: n1=0, n2=2 OOP Development

39. Path expression • Representing in a generic way an execution sequence • Can represent the complete program • Use operators: + for choice . For a concatenation {} * for zero or more repetition {}+ for one or more repetition () for grouping

41. Path expression • Useful to study properties of test sequences • Annotate the transitions with desirable property then analyze the resulting expression • For example: assign a probability to each transition – allows you to evaluate the probability of each path; helps you decide on testing the most probable paths

42. Integration testing • After testing all modules individually using both white and black box testing techniques • Top-down integration • Bottom-up integration • Sandwich / hybrid integration

43. Top down integration • Start with top level modules and integrate left to rigth • Good for testing GUI heavy modules • Increase of confidence to see GUI at the beginning • Less time to test low level utility modules – high reusability modules • Variation: integrate i/o modules to get real input and outputs as early as possible • Need stubs – fake modules to represent called modules

45. Bottom up integration • Start from low level modules • Good testing of low level utility modules • Can see the GUI late in the process • Require driver modules – to call the integrated modules

47. Sandwich integration testing • Two teams – one starting bottom up and the other starting top down • They meet in the middle • Good testing of both high level GUI modules and low level utility modules • Requires coordination and good management to ensure middle level modules also get a good share of the testing time

48. System Testing • Performed in terms of Inputs and outputs of the system. • Performed on the targeted platform. • Goals: • Reveal bugs that are represented only at system level. • Demonstrate that the system implements all required capabilities. • Answer the question: Can we ship it yet? OOP Development

49. Regression Testing • Ensuring that modifications have not caused unintended effects. • Retesting is required OOP Development