Introduction to Software Testing

1. Introduction to Software Testing WISTPC 2010 Peter J. Clarke 06/23/2010

2. Outline of Presentation • Testing terminology • Introduction to Testing Theory • Testing Concepts • Laws of Testing • Levels of Testing • Unit Testing

3. What is software testing? • Software testing is the process of operating software under specified conditions, observing or recording the results and making an evaluation of some aspect of the software. (IEEE/ANSI std 610.12-1990)

4. Testing Terminology The following defns are taken from Binder 2000 and McGregor & Sykes 2001. • Failure – is the manifested inability of a system or component to perform a required function within specified limits e.g. abnormal termination, or unmet time and space constraints of the software. • Fault- incorrect step, process, or data definition in the software. • Error- a human action that produces a fault.

5. Testing Terminology cont • A bug refers to an error or a fault. • Debugging is the process of tracking down the source of failures (errors and/or faults) and making repairs. • A test case specifies the pretest state of the component under test (CUT) and its environment, the test inputs and conditions, and the expected results. • A test point is a specific value for test case input state variables.

6. Testing Terminology cont • A test suite is a collection of test cases, typically related to some testing goal or criterion. • A test driver is a class or utility program that applies test cases to a CUT. • A stub is a partial, temporary implementation of a component. • A test harness is a system of test drivers and other tools to support test execution.

7. Introduction to Testing Theory Basic defns: F is a program, D is the domain and R the output for F. i.e., F: D  R On input d D, F (if it terminates) produces output F(d) R. F is correct on input d (abbreviated OK(d) ) if F(d) exists and OUT(d, F(d)), i.e.,OUT(d, F(d)) is true iffF(d)is an acceptable result. OUT can be considered the specification for F.

8. Introduction to Testing Theory .F(d) d. F(t) t T F R D T constitutes and ideal test if OK(t) for all t T implies OK(d) for all d D.

9. Introduction to Testing Theory cont Basic defns cont: A testT for program F is simply a finite subset of D. If T = D then we have exhaustive testing and the above holds. It is usually impractical to perform exhaustive testing A test is successful iff ( t T) OK(t))

10. Testing Concepts Test Case components: • Name – identifies the test case, it is a good idea to derive the name from the requirement being tested. • Purpose – states the purpose of the test and relates it to the requirement (or scenario). • Test set up – describe the h/w and s/w and environment required for a successful test. • Input – description of the input data or commands. • Expected output(or Oracle) – expected test results against which the output of the test is compared. • Actual output(or log) – output produced by the test.

11. Testing Concepts cont • Test cases are classified depending on which aspect of the system model is tested: • Blackbox (specification-based or functional)tests focus on the input/output behavior (or functionality) of the component. Tests do not deal with the internal aspects of the component nor with the behavior or the structure of the components. • Whitebox (structural or implementation-based) tests focus on the internal structure of the component. That is, test cases are constructed based on the code that implements the software.

12. Testing Concepts cont • Combination of blackbox and whitebox testing is referred to as graybox testing. • Regression testing includes the re-execution of prior tests after a change, this ensures that the functionality that worked before the change has not been affected.

13. Testing Concepts cont • Test criteria - The criteria that a system or component must meet in order to pass a given test. [IEEE 610] There are two types of testing criteria: • Test data selection criterion – represents a rule used to determine which test cases to select. • Test data adequacy criterion – a rule used determine whether or not sufficient testing has been performed.

14. Laws of Testing Handbook of SSE • Testing can show the presence but not the absence of errors (Dijkstra). • A developer is unsuited to test his or her code (Weinberg’s). • Approximately 80 percent of defects come from 20 percent of modules (Paret-Zipf-type) • Suspicion-based testing can be more effective than most other approaches (Hamlet’s hypothesis).

15. Levels of Testing • Unit Testing - refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the minimal unit tests include the constructors and destructors. (wikipedia, 2010) • Integration Testing - is any type of software testing that seeks to verify the interfaces between components against a software design. Components may be integrated in an iterative way or all together ("big bang"). (wikipedia, 2010)

16. Levels of Testing • System Testing - testing a completely integrated system to verify that it meets its requirements. (wikipedia, 2010) See http://en.wikipedia.org/wiki/Software_testing • Let us take a look at the Premium Bank Account (FBA) specification and Java program

17. Unit Testing • Focuses on the building blocks of the software system i.e., objects and subsystems. • Many unit testing techniques have been devised including: equivalence testing, state-based testing, boundary testing, domain testing, control flow-based testing (statement, branch, MCC, basis path, path).

18. Unit Testing – Equivalence Partitioning • Equivalence partitioning is a blackbox testing technique that minimizes the number of test cases. • Possible inputs are partitioned into equivalence testing classes, and a test case is selected from each class. • Assumption - system behaves in a similar way for all members of an equiv. class. • Criteria used to determine equivalence classes: coverage, disjointedness, representation.

19. Example: Valid inputs to test the getNumberDaysInMonth() method Unit Testing – Equivalence Partitioning

20. Unit Testing – Equivalence Partitioning Exercise: • Identify at least two (2) test cases for the FBA program using equivalence testing for part (1) of the specification. • Perform a walkthrough (trace) of the program to determine if the program is correct with respect to the specification. • Repeat steps 1 and 2 for the other parts of the specification i.e., parts 2, 3, 4, 5.

21. Unit Testing – State-based • State-based testing exercises the implementation to see whether it produces the correct response for different transitions in some model of the system. Blackbox testing technique. • A state machine is an abstraction composed of events (inputs), states, and actions (outputs). • A state machine has four building blocks • State – an abstraction that summarizes the information form the past that helps determine the behavior for the future. • Transition – an allowable tow state sequence. A transition is cause by an event. event-name arg-list ‘[‘ guard-condition’]’ ‘/’ action-expr

22. Unit Testing – State-based • Event – An input or an interval of time • Action - The result or output that follows an event. Example of state machine model of Stack with guards Empty pop/EmptyStackException push(x) pop [n == 1] / return top(x) push(x) [n < max-1] Loaded pop [n > 1] / return top(x) push(x) [n == max-1] pop / return top(x) push(x) / FullStackException Full

23. Specification-based Testing cont Test generation for state-based models: • Piecewise coverage - all states, all events, all actions. • All Transitions • All n-Transition Sequences • All Round-trip Paths • M-length Signature

24. Faults Revealed Strategy All possible invalid states All event and action faults Exhaustive All invalid states Up to M extra states All event and action faults M-length Signature All invalid/missing states Some extra states All event and action faults All Round-trip Paths All Transition k-tuples All event and action faults Some missing/invalid states All Transitions Some event and action faults All States All Events All Actions Fig. State-based test power hierarchy

25. Unit Testing – State-based Exercise: • Create a state machine from the FBA specification. • Identify test cases that covers the all states, all events and all transitions criteria. • Identify test cases that covers the all 2-transitions sequences criteria.

26. Unit Testing – Boundary Analysis • Is used in conjunction with equivalence partitioning in that it focuses on a source of faults - the boundaries of an equivalence partition. Blackbox testing technique. • Test data derived by examining the edges of an equivalence partition. • One form of domain testing is based on boundary analysis: • Indentify constraints for all input variables (boundaries) • Select test values for each variable in each boundary (on point)

27. Unit Testing – Boundary Analysis • Select test values that do not lie on any boundary (off point) • Select test values that satisfies all boundaries but do not lie on the boundary (in point). • Select test values that do not satisfy any boundary condition (out point). Stack Example: empty (Stack.size() == 0) loaded (Stack.size() > 0) && Stack.size() < MAXSTACK) full (Stack.size() == MAXSTACK)

28. Unit Testing – Boundary Analysis Assume MAXSTACK = 32767 P184 in Testing Object-Oriented Systems by Binder.

29. Unit Testing – Boundary Analysis Exercise: • Identify test cases for the FBA program using boundary analysis (in, on, off) for part (1) of the specification. • Perform a walkthrough (trace) of the program to determine if the program is correct with respect to the specification. • Repeat steps 1 and 2 for the other parts of the specification i.e., parts 2, 3, 4, 5.

30. Unit Testing – Domain Analysis • Domain testing is a blackbox testing technique that can be used to identify efficient and effective test cases when multiple variables can or should be tested together. • It builds on and generalizes equivalence class and boundary value testing to n simultaneous dimensions. Like those techniques the tester is searching for situations where the boundary has been defined or implemented incorrectly. • Think about the solutions for an n-dimensional space.

31. Control Flow Adequacy Criteria Flow Graph (FG) [Binder 00] • A segment is represented by a node in the FG (a sphere). A segment is one or more contiguous statements with no conditionally executed statements. • A conditional transfer of control is a branch. A branch is represented by an outgoing edge in the FG. • The entry point of a method is represented by the entry node, which is a node no incoming edges. The exit point of a method is represented by the exit node, which has no outbound edges.

32. Control Flow Adequacy Criteria cont Flow graph Entry A Example: Source code public int Fun( int x, int K){ while (x <= 10 && k < 3){ if (x%2 != 0) k = k + 1; x = x + 1; } if (x < 0){ x = 10; k = 0; } return k; } B K=0 F x<=10 && k<3 C G T F X<0 F x%2 != 0 D T T X = 10 K = 0 k=k+1 E H x=x+1 F return k I

33. Unit Testing – Statement Coverage • Statement coverage – A set P of execution paths satisfies the statement coverage criterion iff for all nodes n in the FG, there is at least one path p in P s.t. n is on the path p. Whitebox testing technique. • Generate test data to execute every stmt in the program at least once. • Give a value(s) of x and k in the example that satisfies statement coverage.

34. Unit Testing – Branch Coverage • Branch coverage – A set P of execution paths satisfies the branch coverage criterion iff for all edges e in the FG, there is at least one path p in P s.t. p contains edge e. Whitebox testing technique. • Generate test data to exercise the true and false outcomes of every decision. • Give a value(s) of x and k in the example that satisfies branch coverage.

35. Unit Testing – MCC Coverage • Multiple Condition Coverage (MCC) – A test set T is said to be adequate according to the multiple-condition-coverage criterion if, for every condition C, which consists of atomic predicates (p1, p2, …,pn), and all the possible combinations (b1, b2, …,bn) of their truth values, there is al least one test case in T s.t. the value of pi equals bi, i= 1, 2, …,n. Whitebox testing technique.

36. Unit Testing – MCC Coverage cont • Generate test data to exercise all possible combinations of true and false outcomes of conditions within a decision. • Give a value(s) of x and k in the example that satisfies MCC coverage.

37. Unit Testing – Path Coverage • Path coverage – A set P of execution paths satisfies the path coverage criterion iff P contains all execution paths from the begin node to the end node in the FG. Whitebox testing technique. • Generate test data to exercise all paths in the program. Note every path through a loop is considered to be different and distinct. • Give a value(s) of x and k in the example that satisfies path coverage.

38. Unit Testing – Basis Path Coverage • Cyclomatic-Number Criterion – A set P of execution paths satisfies the cyclomatic number criterion iff P contains at least one set of v independent paths, where v = e – n + p is the cyclomatic number of the FG. • n - number of vertices, e - is the number of edges, p - is the number of strongly connected components. • A graph is strongly connected if, for any two nodes a and b there exists a path from a to b and a path from b to a.

39. Unit Testing – Basis Path Coverage • Basis path testing – is a way of easily identifying the upper bound for the number of paths necessary in order to achieve branch coverage. Whitebox testing technique. v = (e + 1) - n + 1 McCabe Cycolmatic Complexity Number • Note this is for a strongly connected FG, add edge from exit node to entry node, hence e + 1. • Note some formulae include the extra edge in the “e”. • Identify the basis set for the FG example.

40. Control Flow Adequacy Criteria cont

41. Control Flow Adequacy Criteria cont Points to note: • Path coverage not always practical due to loops i.e., infinite number of paths. • Statement and branch coverage are not finitely applicable due to dead code and infeasible branches. • Can define a finitely applicable version of the criteria.

42. Unit Testing – Whitebox testing Exercise for each whitebox testing technique: Using all the blackbox test cases developed for the PBA problems perform the following. • Build the FG for each method and perform a walkthrough to check if the coverage criteria is satisfied. • If the criteria is not satisfied add test cases to ensure the coverage criteria is satisfied.

43. Summary • Testing terminology – software testing, test case, test driver, stub, fault, failure, error • Introduction to Testing Theory – relationship to functions and the concept of an ideal test. • Testing Concepts – blackbox, whitebox, graybox testing • Laws of Testing • Levels of Testing – unit, integration, system

44. Summary cont • Unit Testing • Blackbox: equivalence partitioning, state-based, boundary analysis, domain analysis. • Whitebox: statement coverage, branch coverage, multiple-condition coverage, path coverage, basis path coverage. • Practice exercises were given for each testing technique.