Basic definitions

1. Basic definitions • A failure is an unacceptable behaviour exhibited by a system • The frequency of failures measures the reliability • An important design objective is to achieve a very low failure rate and hence high reliability • A failure can result from a violation of an explicit or implicit requirement • A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures • could be in the requirements, the design and the code • It might take several defects to cause a particular failure • An error is a slip-up or inappropriate decision by a software developer that leads to the introduction of a defect into the system Lecture 9

2. Exercise Categorize the following according to whether each describes a failure, a defect or an error: • A software engineer working in a hurry, unintentionally deletes an important line of source code • On 1 January 2040 the system reports the date is January 1940 • No design documentation or source code comments are provided for a complex algorithm • A fixed size array of length 10 is used to maintain the list of courses taken by a student during one semester. The requirements are silent about the maximum number of courses a student may take at any one time Lecture 9

3. Inspections • An inspection is an activity in which one or more people systematically • Examine source code or documentation, looking for defects. • Normally, inspection involves a meeting... • Although participants can also inspect alone at their desks Lecture 9

4. Roles on inspection teams • The author • The moderator • Calls and runs the meeting • Makes sure that the general principles of inspection are adhered to • The secretary • Responsible for recording the defects when they are found • Must have a thorough knowledge of software engineering • Paraphrasers • Step through the document explaining it in their own words Lecture 9

5. Principles of inspecting • Inspect the most important documents of all types • code, design documents, test plans and requirements • Choose an effective and efficient inspection team • between two and five people • Including experienced software engineers • Require that participants prepare for inspections • They should study the documents prior to the meeting and come prepared with a list of defects • Only inspect documents that are ready • Attempting to inspect a very poor document will result in defects being missed Lecture 9

6. Principles of inspecting • Avoid discussing how to fix defects • Fixing defects can be left to the author • Avoid discussing style issues • Issues like are important, but should be discussed separately • Do not rush the inspection process • A good speed to inspect is • 200 lines of code per hour (including comments) • or ten pages of text per hour Lecture 9

7. Principles of inspecting • Avoid making participants tired • It is best not to inspect for more than two hours at a time, or for more than four hours a day • Keep and use logs of inspections • You can also use the logs to track the quality of the design process • Re-inspect when changes are made • You should re-inspect any document or code that is changed more than 20% Lecture 9

8. A peer-review process • Managers are normally not involved • This allows the participants to express their criticisms more openly, not fearing repercussions • The members of an inspection team should feel they are all working together to create a better document • Nobody should be blamed Lecture 9

9. Conducting an inspection meeting 1. The moderator calls the meeting and distributes the documents 2. The participants prepare for the meeting in advance 3. At the start of the meeting, the moderator explains the procedures and verifies that everybody has prepared 4. Paraphrasers take turns explaining the contents of the document or code, without reading it verbatim • Requiring that the paraphraser not be the author ensures that the paraphraser say what he or she sees, not what the author intended to say 5. Everybody speaks up when they notice a defect Lecture 9

10. Software Testing • Definition: Testing is a process of executing a program with data with the sole intention to find errors in the program • Can only reveal the PRESENCE of ERRORS, and NOT the ABSENCE of ERRORS • A successful test is a one which discovers more, yet-undiscovered, errors • Consequence: • An ideal testing process will never terminate Lecture 9

11. Effective and Efficient Testing • To test effectively, you must use a strategy that uncovers as many defects as possible. • To test efficiently, you must find the largest possible number of defects using the fewest possible tests • Testing is like detective work: • The tester must try to understand how programmers and designers think, so as to better find defects • The tester must not leave anything uncovered, and must be suspicious of everything • It does not pay to take an excessive amount of time; tester has to be efficient Lecture 9

12. Objectives of testing • To find more errors in the design and implementation of a software product • To conform that the product meets (satisfies) the requirements • To conform performance requirements • Testing is one type of validation • Validation is a process that answers to the question • “Are we developing the right product?” • Consequence: The product must be ready (available) in some form before exercising the testing process Lecture 9

13. Two Types • A fault of commission: representation is incorrect • A fault of omission: fail to enter correct information, example: missing a requirement fault of commission Specification fault of omissions Correct Code Implementation Lecture 9

14. Specification (expected) Program (observed) S P 6 5 2 1 3 4 7 T Testing Relationship Lecture 9

15. Testing and Debugging • Definitions: • Testing is a process of finding the presence of errors • Debugging is a process of finding the location and cause of errors • Debugging requires testing efforts to find the errors first, and then retest the program after correcting the errors • Thus, testing occurs before and after debugging • Testing may be done manually, but debugging will be done effectively with the help of tool support Lecture 9

16. Testing Process • Testing activities: • test planning • test case development • running test cases • evaluating test results Lecture 9

17. Test case • Definition: A test case is a description of input/output relationship of one particular testing activity • Example: • Let y = f(x) be the function to be tested; a test case will indicate for what values (range of values) of x the function will promptly return an output, and for what values the function will not return an output • Generally expressed as a set of predicates (boolean expressions) Lecture 9

18. Test Data • Definition: Test data is a set of values that are used to test a program • Example: • In the y = f(x) example, test data could be x = 12.34, x = 14.56, x = -0.3487 etc. • Test data are generally derived from the application domain, particularly from the requirements Lecture 9

19. Test Case Information Test Case ID Purpose Preconditions Inputs Expected Outputs Postconditions Execution History Date Result Version Run By Lecture 9

20. Testing principles • All tests should be traceable to requirements • Otherwise, testing efforts will not be useful • Testing activity should be planned well before testing process is exercised • Requires preparation of test cases, test data and test oracles • Exhaustive testing is not possible • Often, testers are required to select (may be randomly) a subset of test cases to exercise • To be most effective, testing should be conducted by a third party – not by the developers • Quality Assurance team Lecture 9

21. Test case design methods • Definitions: • Black-box testing: Program is assumed to be a black-box and test cases are derived from sources other than the program itself; such sources can be requirements document, design document, problem description etc. • White-box testing: Test cases are derived from the actual source code of the program • Both black-box and white-box testing are required for a successful testing process Lecture 9

22. Test cases can be derived before the program is written Objective is to ensure that the program meets (satisfies) the requirements Needs knowledge of the application domain but nothing about the program or the programming language Test cases can be derived only after the program is complete Objective is to ensure that the logic of the program is correct Needs knowledge of the programming knowledge, and may not need application domain information Black-box and White-box testing Lecture 9

23. White-box testing • Also called “glass-box testing” and “structural testing” • Derivation of test cases from the program (source code); knowledge of the application domain may be used to derive additional test cases Lecture 9

24. White-box Testing • Identify test cases based on how the program is actually implemented • Test coverage metrics • Provide a way to explicitly state the extent to which a software item has been tested • Make the testing management more meaningful Lecture 9

25. Test Coverage Metrics • Goal • Derive sufficient number of test cases to cover one or more of the following: • Statement Coverage • Branch Coverage / Condition Coverage • Path Coverage Lecture 9

26. Example – computing factorial int factorial (int n) { inti, ret = 0; if (n < 0) { return ret; } i = n; ret = 1; while (i > 0) { ret = ret * i; i = i - 1; } return ret; } Lecture 9

27. Example – computing factorial (continued) • Test cases: • case 1: n is negative; return value must be 0 • case 2: n is positive; return value must be n! • Test data: • n = -2, n = 5 • Test oracles • run the program for the above test data, get the actual outputs and check against the expected outputs shows in test cases Lecture 9

28. Is Statement Coverage Sufficient? • Example: • Consider the following part of the code that is supposed to compute the absolute value of y: • if (y >= 0) • then y = 0 - y; • abs = y; Lecture 9

29. Branch Coverage / Condition Coverage • Branch coverage: • focus on testing branches in code • Test cases are selected in such a way that each branch in the code is executed at least once. • This requires that each decision box is evaluated to be true and false at least once. Lecture 9

30. Branch Coverage / Condition Coverage • Example: • If ((x < -5 ) && (y > 11)) • z = compute (x,y); • else z = compute_altern (x,y); • Consider the test cases: • x = -4, y = 10 and x = -6 , y = 12 Lecture 9

31. X> 0 No Yes z = someF2(x) z = someF1(x) z > 10 Yes No z = doSth1(); z = doSth2(); Another Example • Consider: • Test cases: • x = 2, z = 6 • x = 0, z = 12 Lecture 9

32. Path coverage • Path coverage: considers all possible logical paths in a program and leads to test cases aimed at exercising a program along a path. • the use of the path coverage criterion may help detect faults easily omitted by branch coverage testing. • However, in many cases, this criterion can be too impractical, especially when it comes to loops in the program that may easily lead to a very high number of paths. Lecture 9

33. Loop Coverage • Loop Coverage • Ensure that every loop is covered for all possible execution scenarios • Loop skipped • Loop is iterated through at the first time • Loop is iterated through many times • Loop is iterated at its last time • Independent loops are easier to test than nested loops Lecture 9

34. Flow graph notation • Also called “program graph” • Describes the logical control flow in the program • Consists of nodes (represented as circles) and transitions (represented by arrows) • nodes represent executable statements or expressions • transitions represent control flow Lecture 9

35. sequence Do-while if if-else while Flow graph notations Switch-case for Lecture 9

36. Try-catch example Line1intage; Line2 try { Line3System.out.print(“Enter your age: “); Line4 age = Keyboard.readInt(); Line5 if(age < 0 || age > 120) { Line6 throw new Exception(“Invalid age: “ + age); } Line7System.out.println(“Your age is ” + age); } Line8 catch (Exception e) { Line9System.out.println(e.getMessage()); } Line10System.out.println(“Done.”); Lecture 9

37. Flow graph for the factorial example 1 ret = 0 3 2 n < 0 return ret Notice that this is a special case; “return” is a terminal statement and so the arrow goes out rather than connecting back to the next statement 4 i = n; ret = 1 5 while (i > 0) ret = ret * i; i = i - 1 6 7 return ret Lecture 9

38. Test cases: Case 1: n < 0 will cover statements 1, 2, 3 and return 0; Case 2: n = 0 will cover statements 1, 2, 4, 5, 7 and return 1. Case 3: n > 0 will cover statements 1, 2, 4, 5, 6, 7 and return n! All statements are executed at least once with these test cases. Lecture 9

39. Another example To remove an integer from an array and squeeze the array; add 0 at the end void squeeze (int k) { boolean flag = false; line1 int j, index = 0; line2 while (index < arr.length && !flag) { line3 if (arr[index] == k) { line4 for (j = index; j < arr.length-1; j++) line5 arr[j] = arr[j+1]; line6 arr[arr.length-1] = 0; line7 flag = true; line8 } else index = index + 1; line9 } } Draw the flow graph for this example Identify representative path/independent path Lecture 9

40. Cyclomatic complexity • Definition: Cyclomatic complexity is a software metric (quantitative) that denotes the number of independent paths derived from the flow graph • Definition: An independent path in a flow graph is any path that introduces at least one new set of processing statements or a new conditional expression • In terms of flow graph, there must be at least one new edge that is not traversed. • Example: for the factorial method, it is 3. Lecture 9

41. Computing cyclomatic complexity • Gives a measure of number of test cases to be considered • Cyclomatic complexity V(G) of a flow graph G is defined as • V(G) = E – N + 1 where ‘E’ denotes number of edges, and ‘N’ denotes number of nodes Check this formula for the ‘factorial’ and ‘squeeze’ methods Lecture 9

42. Exercise • Cyclomaticcomplexity differences Lecture 9

43. Specification Program Specification Program Test Cases (Method A) Test Cases (Method B) Comparing White-box testing methods Lecture 9

44. Questions • Are branch and path coverage the same? • Is 100% coverage the same as exhaustive testing? • When can I stop testing? Lecture 9

45. Black-box testing • Testers provide the system with inputs and observe the outputs • They can see none of: • The source code • The internal data • Any of the design documentation describing the system’s internals Lecture 9

46. Example for black-box testing • An automated teller machine (ATM) • consider the functional requirement “withdraw” • test cases for this function will include • withdraw an amount within the balance • withdraw an amount equal to the balance • withdraw an amount more than the balance ???“withdraw a negative amount (though not applicable in practice, you can do so in software)” • no matter how the ‘withdraw’ function is implemented, these test cases will effectively test whether the implementation is satisfactory with regard to the requirements Lecture 9

47. Objectives of black-box testing • Identify incorrect or missing functionalities • Identify errors in interfaces among the components of the software • Identify errors in links with other programs such as databases • a special case of the previous objective • Identify performance errors • Identify initialization and/or termination errors Lecture 9

48. Equivalence partitioning • A method to generate test data for black-box testing • Divides the input domain (the set of values that can be used as input) into mutually disjoint subsets • these subsets are called ‘equivalence classes’ • Each member in a subset has the same characteristic as that of the other elements in the same subset • consequence: it is sufficient to select one data from each subset  reduce the time for testing Lecture 9

49. Mathematical Definition of Equivalence Classes • Given a set S, the equivalence classes of S are the subsets S1, S2, …, Sn such that • S1 S2 … Sn=S • SiSj = {}, for 1<i, j<=n, i j • The first condition asserts that the subsets are partitions of the given set S • The second condition asserts that the subsets are mutually exclusive Lecture 9

50. Input domain Software system Important: Equivalence partitioning does not impose any constraints on the output domain Output domain Lecture 9