Functional System Testing

1. Functional System Testing Written by Adam Carmi

2. Outline • Goal of testing • Test cases, test suites and test data • What is functional system testing? • Coverage • Functional testing techniques: • Functional analysis • Equivalence partitioning • Boundary value analysis

3. The goal of software testing • The process of uncovering evidence of defects in software systems • Does not include efforts associated with tracking down bugs and fixing them • No amount of testing will improve the quality of a computer program • The more testing we do of a system, the more convinced we might be of its correctness • Testing cannot in general prove a system works 100% correctly

4. Test cases • The basic component of testing is a Test Case • In its most general form: (inputs, expected-result) • inputs include system state, user commands and data values to be processed • expected result includes visible/audible interface changes or changes in the system state • Test cases are organized into Test Suites • functionality, security, performance, …

5. Test case execution • A running of the software (under test) that provides the inputs specified in the test case and observing the results and comparing them to those specified by the test case • If the actual result varies from the expected result, then a failure has been detected

6. Test data • An effective test strategy requires careful acquisition and preparation of test data prior to testing • Testing can suffer if test data is poor • Test data concerns: • Depth: quantity and size of data • Breadth: variance of data values and data types • Scope: completeness, relevance and accuracy of data • Result of a query should be valid for the specific purpose of the query, and not due to a missing or inappropriate value • Conditions: data should reflect specific “conditions” in the domain • Data that would otherwise arrive after performing specific operations over time • Test data and test results are expensive to construct

7. Example: Test data for TVRS

8. Specification vs. implementation • The basic approaches to testing software are based on its specification and implementation • White box testing – test cases and data are constructed based on the code that implements the software • quality and correctness of computations is validated • will not be further discussed in this tutorial • Black box testing – test cases and data are constructed based solely on the software’s specification • Gray box testing – test cases are constructed to directly target various modules and layers of the system (requires architectural insight)

9. Functional System Testing • Testing of a completed application to determine that it provides all of the behaviors required of it • Testing of completed increments that provide some degree of end-user functionality • Search for defects that are variances between the actual operation of the system and the requirements for the system • System is treated as a black/gray box

10. How much testing is adequate? • Completely validating IEEE 754 floating-point division requires 264 test-cases! float divide(float x, float y) • From practical and economic perspectives, exhaustive testing is usually not possible • Which software pieces should we test? • Which test cases should we choose?

11. Coverage • Coverage is a measure of how completely a test suite exercises the capabilities of a piece of software • “Each line of code should be executed at least once” • “One test case should be constructed from each specified requirement” • It is necessary to use testing techniques that narrow down the number of test cases allowing the broadest testing coverage with the least effort

12. Technique: Functional Analysis • Analyze the expected behavior of the system according to its functional specification • Generate a test procedure for each of the possible usage scenarios • Corresponds to use case scenarios • Analyze how a change in one part of the system affects other parts • “Grand tour” test cases: the result of one test case produces the data that is the input to the next test case

13. Example: Functional Analysis I Use Case: Remove Traffic Violation • Supervisor calls for deletion of the chosen Traffic Violation • TVRS prompts Supervisor for confirmation • Supervisor confirms • TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record • OffendersDB approves that the Traffic Violation has been deleted • TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC

14. Example: Functional Analysis II

15. Example: Functional Analysis III How do we know that violation 243567 is stored in the system? In addition, a query could be run on the Offenders database Verify effects of change Filled when the test case is executed Can a tester diagnose the cause of a defect?

16. Technique: Equivalence Partitioning • Identifies ranges of input and initial conditions that are expected to produce the same result • A group of test cases form an equivalence class if: • They test the same feature/scenario • If one test reveals a fault, the other ones (probably) will too • If a test does not reveal a fault, the other ones (probably) will not either • It is adequate to use only a single representative of the equivalence class

17. Example: Equivalence Partitioning I Input value specification for “Lookup Violation” form:

18. Example: Equivalence Partitioning II

19. Example: Equivalence Partitioning III • The number of test cases to choose from is reduced to(3 + 3) × (4 + 2) × (4 + 2) = 216 • The actual number can be further limited • Single invalid field per test case (3 × 4 × 4 + 7 = 55) • Importance of use case • Resources available • Most frequent input • Life-critical software • Infeasible test cases • Randomly • ...

20. Technique: Boundary Value Analysis • Based on experience / heuristics • Testing boundary conditions of equivalence classes is more effective • Choose input boundary values as equivalence classes representatives • Choose inputs that invoke output boundary values • Examples: • (0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11 • Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements

21. BVA as an equivalence partitioning extension • Choose one (or more) arbitrary value(s) in each equivalence class • Choose valid values exactly on lower and upper boundaries of equivalence class • Choose invalid values immediately below and above each boundary (if applicable)

22. General purpose test-suite construction technique • May be used to obtain reasonable coverage with little effort • Use cautiously! • Unsuitable when values of different fields are related • While test cases can be added • Each new test case should include as many un-included valid non-boundary equivalence class representatives as possible • While test cases can be added • Each new test case should include as many un-included valid boundary equivalence class representatives as possible • While test cases can be added • Each new test case should include a single invalid equivalence class representative that has not been included before • Manually replace/remove redundant or infeasible test-cases

23. Example: Country Club I Specification

24. Example: Country Club II A combo box is used for choosing the day and guest status

25. Example: Country Club III valid valid (boundary) invalid