Introduction to Software Testing

1. Introduction to Software Testing

2. Reading Assignment • P. Ammann and J. Offutt “Introduction to Software Testing” • Chapter 1

3. Outline • What is Software Testing? • What is the cost of software failures? • Observability and controllability • Black-box & White-box Testing • Changing notions of testing

4. What is Software Testing? • The process of finding evidence of defects in software systems. • Establishing confidence that a program does what it is supposed to do. Software testing is not debugging. Software testing is not quality assurance

5. Software Testing Vs Quality Assurance (QA) • Testing is necessary but not enough for QA process. • Testing contributes to improve quality by helping to identify problems. • QA sets standards that project members (including testers) should follow in order to build a better software.

6. What is Software Testing? • Correctness of software with respect to requirements or intent; • Performance of software under various conditions; • Robustness of software, its ability to handle erroneous input and unanticipated conditions; • Installation and other facets of a software release.

7. Test Engineer & Test Managers • Test Engineer : An IT professional who is in charge of one or more technical test activities • designing test inputs • producing test values • running test scripts • analyzing results • reporting results to developers and managers

8. Test Engineer & Test Managers • Test Manager : In charge of one or more test engineers • sets test policies and processes • interacts with other managers on the project • otherwise helps the engineers do their work

9. P Computer Evaluate Test Engineer Activities Test Manager Test Designs Executable Tests instantiate design Test Engineer Test Engineer execute Output

10. Basic Definitions • Software Fault • A static defect in the software. • Software Error • An incorrect internal state that is the manifestation of some fault. • Software Failure • External, incorrect behavior with respect to the requirements or other description of the expected behavior. Faults in software are design mistakes and will always exist

11. Example numZero([0,5,0])=1 public static int numZero (int[] x) { // Effects: if x == null throw NullPointerException // else return the number of occurrences of 0 in x int count = 0; for (int i = 1; i < x.length; i++) { if (x[i] == 0) { count++; } } return count; } Fault Failure ([4,5,0], 0, 1, “if”) ([0,5,0], 0, 1, “if”) Error that does not result in a failure What is the state at the if statement on the very first iteration of the loop (x,count,i,PC)? Error

12. Important Considerations • Detect system failures by choosing test inputs carefully. • Determine the faults leading to the failures detected. • Repair the faults leading to the failures detected; and • Re-test the module/system.

13. Why do Failure Occur?

14. Root Causes of Failures • Inaccurate understanding of end user requirements. • Inability to deal with changing requirements. • Late discovery of serious project flaws. • For example, modules that do not fit together. • Untrustworthy build & release process. • Implementation team’s chaos.

15. Failure Costs Disney’s Lion King CD

16. Observability and Controllability • Software Observability – How easy it is to observe the behavior of a program in terms of its • Outputs, effects on the environment and • Other hardware and software components. Software that affects hardware devices, databases have Low observability.

17. Observability and Controllability • Software Controllability – How easy it is to provide a program with the needed inputs, in terms of values, operations and behavior. • Easy to control software with inputs from keyboards. • Input from hardware sensors is harder.

18. Inputs to affect controllability and observability • Prefix Values: • Any inputs necessary to put the software into the appropriate state to receive the test case values. • Post-Fix Values: • Any inputs that need to be sent to the software after the test cases values.

19. Inputs to affect controllability and observability • Two Types of postfix values • Verification Values: Values necessary to see the results of the test case values. • Exit Commands: Values needed to terminate the program or otherwise return it to a stable state. • Execution Test Script • A test case that is prepared in a form to be executed automatically on the test software and produce a report.

20. Top-Down and Bottom-up Testing • Top-Down Testing: Test the main procedure, • Then go down through procedures its calls, and so on. • Bottom-up Testing: Test the leaves in the tree (procedures that make no calls), and move up to the root. • Each procedure is not tested until all of its children have been tested.

21. requirements output input events Black-Box Testing

22. Black-box Testing • Test cases are derived from formal specification of the system. • Test case selection can be done without any reference to the program design or code. • Only tests the functionality and features of the program. • Not the internal operation.

23. Black-box Testing • Advantages • Test case selection is done before the implementation of a program. • Help in getting the design and coding correct with respect to the specification.

24. White-Box Testing

25. White-Box Testing • Test cases are derived from the internal design specification or actual code for the program. • Advantages • Tests the internal details of the code; • Checks all paths that a program can execute. • Limitations • Wait until after designing and coding the program under test in order to select test cases.

26. Changing Notions of Testing • Traditional View of testing is of testing at specific software development phases • E.g. unit, module, integration etc. • New view is in terms of structures and criteria. • Graphs, logical expressions, syntax, input spaces.

27. main Class P Class B Class A • System testing: Test the overall functionality of the system method mA1() method mB1() • Integration testing: Test how modules interact with each other method mA2() method mB2() • Module testing: Test each class, file, module or component • Unit testing: Test each unit (method) individually Traditional Testing at different levels • Acceptance testing: Is the software acceptable to the user?

28. New Approach: Test Coverage Criteria • Define a model of the software, then find ways to cover it. • Test requirements: Specific things that must be satisfied or covered during testing. • Test criteria: A collection of rules and a process that define test requirements.

29. Key Points • Software testing is the process of discovering evidence of defects and failures in software systems. • Test early, test often, test enough. • Testers should have good understanding of the development process, product.

30. Key Points • Testers should have good understanding of the development process, product. • Inputs to affect controllability and observability • New testing view is in terms of structures and criteria. • Graphs, logical expressions, syntax, input spaces.