CIS-74 Computer Software Quality Assurance

1. CIS-74 Computer Software Quality Assurance Systematic Software Testing Chapter 5: Analysis and Design

2. Creating an Inventory 1. Gather reference materials 2. Form a brainstorming team 3. Determine test objectives 4. Prioritize objectives <continued on next slide>

3. Creating an Inventory 5. Parse (organize) objectives into lists. • Create an inventory tracking matrix • Left-hand column of objectives should be in prioritized order from step 4. • This step known as requirements traceability. <continued on next slide>

4. Creating an Inventory 7. Identify tests for unaddressed conditions, adding/modifying test cases as needed. • Evaluate each inventory item for adequacy. (And add/modify test cases as needed.) • Maintain the testing matrix.

5. Black-Box vs. White-Box Black-box testing - testing done without knowledge of the internal workings of the program/system being tested. White-box testing - testing done with knowledge of the internal workings of the program/system being tested. White-box testing is also called structural testing because it’s based on structure of the code.

6. Black-Box Science: Equivalence Partitioning An equivalence class is formed by a group of tests which: • All test the same thing. • If one catches a bug, the others do too. • If one doesn’t catch a bug, the others won’t either.

7. Black-Box Science: Boundary Value Analysis Test for: • Exact boundaries • Value immediately above upper boundary • Value immediately below lower boundary

8. Black-Box Science: State-Transition Diagrams A state machine is a thing which depends not just on current input, but also on past input, for determining output. A transition occurs when a state is changed by its input from one state into another. To use in black-box testing: Create a state-transition diagram. Create a test case for every arrow (transition) from the diagram.

9. Black-Box Science: Orthogonal Arrays An Lx(Yz) orthogonal array is a two-dimensional array with: • X rows • Z columns • 1..y in cells of matrix

10. Black-Box Science: Orthogonal Arrays • Useful with compatibility testing in particular • What text did not say: • How to create an orthogonal array • How to use an orthogonal array when the variables do not all have the same number of values (as did the example in Table 5-9)

11. Black-Box Art: Ad Hoc Testing • Often based on systematic techniques • Test cases should be created for all bugs found • Can locate an entire class of tests missed using more formal means • Fun!

12. Black-Box Art: Random Testing • All data fields generated randomly, often with a tool • Can lead to too many negative tests - those with an invalid input, and expected results of an error message • Not very useful • At best, useful for “crash-proofing”

13. Black-Box Art: Semi-Random Testing • Random combinations of legitimate values for variables • Gives a little added confidence to systematic testing

14. Black-Box Art: Exploratory Testing • Test design and execution occur concurrently • Test results may prompt user to delve deeper into that area • Neither ad hoc nor random in nature • Productive areas of test are immediately expanded • Viewed as complementary to structured testing by authors

15. White-Box Science: White-Box Testing • “White-box” should be “clear-box” or “glass-box” in authors’ views. (Why?) • Normally viewed as a technique for unit testing, but can also be used with integration & system testing • Primarily about path testing (paths through the code)

16. White-Box Science: Coverage Techniques • Statement coverage - # of lines of code executed by the tests • Decision or branch coverage - # of decisions executed by the tests • Path coverage - # of paths executed by the tests

17. Test Design Documentation:IEEE Test Design Specification • One TDS per feature from the “Features to be Tested” from the corresponding test plan • Test identification section includes test case id’s and high-level description for each test case but not info on how to execute them.

18. Test Design Documentation:IEEE Test Design Specification • Test cases may not be sequential because TDS covers all tests necessary to test one feature--some tests for other features may be reusable. • Feature pass/fail criteria establish success or failure for the TDS’s feature--not individual test cases.

19. Test Design Documentation:Test Case Specification Two approaches: • IEEE Template for Test Case Specification • Focuses on the “what” • Spreadsheet • Very common method • Can also be used to record results

20. Test Design Documentation:IEEE Test Procedure Specification Focuses on “how” tests will be run Can be run via scripts--automated test procedures. These “scripts” are normally written in a scripting language such as Perl or a UNIX/Linux shell language. “Scripts” also refer to testing instructions for manually-executed tests.

21. Review Questions

22. Which test design technique was covered in depth by the first part of Chapter 5?

23. Which test design technique was covered in depth by the first part of Chapter 5? Inventories

24. What are the column headers in an inventory tracking matrix?

25. What are the column headers in an inventory tracking matrix? Test case identifiers.

26. How are the objectives/inventory items in the column of an inventory tracking matrix ordered?

27. How are the objectives/inventory items in the column of an inventory tracking matrix ordered? Priority order (highest to lowest)

28. With respect to an inventory tracking matrix, what is the name for the process of adding “X”s to those non-header cells which represent test case coverage of an objective/inventory-item?

29. With respect to an inventory tracking matrix, what is the name for the process of adding “X”s to those non-header cells which represent test case coverage of an objective/inventory-item? Requirements traceability

30. What is the name for testing based upon the requirements, i.e., the internal workings of the system under test are unknown?

31. What is the name for testing based upon the requirements, i.e., the internal workings of the system under test are unknown? Black-box testing

32. What is another name for black-box testing?

33. What is another name for black-box testing? Behavioral testing.

34. What is white-box testing?

35. What is white-box testing? Testing in which the internal workings of the system under test are known to the tester.

36. What is another name for white-box testing?

37. What is another name for white-box testing? Structural testing. (Why?) Glass-box testing Clear-box testing Translucent-box testing

38. What is coincidental correctness?

39. What is coincidental correctness? Right results for wrong reason. (A test case passes even though the input was processed incorrectly.)

40. What are the three defining aspects of a group of tests making up an equivalence class?

41. What are the three defining aspects of a group of tests making up an equivalence class? They all test the same thing If one catches a bug, the others will too. If one fails to catch a bug, the others will also fail.

42. What are the three areas to test when using boundary analysis?

43. What are the three areas to test when using boundary analysis? The exact boundaries The value immediately above the upper boundary The value immediately below the lower boundary

44. Which black-box science technique involves creating a table which lists all possible inputs and outputs?

45. Which black-box science technique involves creating a table which lists all possible inputs and outputs? Decision tables

46. In a state-transition diagram, how are the states and transitions depicted?

47. In a state-transition diagram, how are the states and transitions depicted? State - circle with label inside Transition - directed arc between two states

48. What is an orthogonal array?

49. What is an orthogonal array? A two-dimensional array of integers Any two columns contain all combinations of the integers contained in the array

50. What is meant by L9(34)?