Unit Testing

1. Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3rd edition, John Wiley & Sons, 2008. Chapter 13.

2. Hierarchy of Testing Testing Ad Hoc Program Testing System Testing Acceptance Testing Unit Testing Integration Testing Function Benchmark Properties Black Box Black Box Black Box Black Box Black Box Black Box Top Down Pilot Performance Equivalence Equivalence Equivalence Equivalence Equivalence Equivalence Bottom Up Reliability Alpha Boundary Boundary Boundary Boundary Boundary Big Bang Availability Decision Table Decision Table Beta Sandwich Security State Transition State Transition State Transition State Transition State Transition Usability Use Case Use Case Use Case Use Case Documentation Domain Analysis Domain Analysis Domain Analysis Portability White Box White Box Capacity Control Flow Data Flow

3. Outline • Basic concepts • Black box testing • White box testing

4. Basic Concepts What is unit testing? Test focusing on the smallest units of code, such as Functions, procedures, subroutines, subprograms Methods, classes Component tested in isolation from the rest of the system and in a controlled environment: Uses appropriately chosen input data Uses component-level design description as guide Often the target of testing frameworks such as JUnit

5. Basics Why unit testing? Foundation for other testing like integration and system testing What to test? Data transformations across the unit are tested Data structures are tested to ensure data integrity When and who? Frequently done (at least partially) during code development by code developers

6. Unit Test Procedures Driver Module to Be Tested Results Test cases Stub Stub

7. Two Different Techniques Black box Based on specification Inner structure of test object is not considered White box Based on source code Inner structure of test object is the basis of test case selection Often complementary Effectiveness of black box is similar to white box, but the mistakes found are different (Hetzel 1976, Myers 1978) Use in combinations

8. Outline Basic concepts Black box testing Basic concepts Equivalence classes … White box testing

9. What Is Black Box Testing? Unit (code, module) seen as a black box No access to the internal or logical structure Determine if given input produces expected output Output Input

10. Black Box Testing Test set is derived from specifications or requirements Goal is to cover the input space Lots of approaches to describing input space: Equivalence classes Boundary value analysis Decision tables State transitions Use cases . . .

11. Advantage and Disadvantage (Dis)Advantages It does not require access to the internal logic of a component However, in most real-world applications, impossible to test all possible inputs Need to define an efficient strategy to limit number of test cases

12. General Process Analyze specifications or requirements Select valid and invalid inputs (i.e., positive and negative tests) Determine expected outputs Construct tests Run tests Compare actual outputs to expected outputs

13. Outline Basic concepts Black box testing Basic concepts Equivalence classes Boundary values … White box testing

14. Motivation Assume you test a sign function, int sign(int x), whose result is defined as: 1 if x > 0 0 if x = 0 -1 otherwise (i.e., x < 0) One way to reduce the number of test cases is: to partition the input into three subsets { x | x > 0}, { x | x = 0}, { x | x < 0} to pick one from each subset, e.g., 10, 0, and -10.

15. Basic Strategy of Equivalence Classes Partition the input into equivalence classes This is the tricky part. It’s an equivalence class if: Every test using one element of the class tests the same thing that any other element of the class tests If an error is found with one element, it should be found with any element If an error is not found with some element, it is not found by any element Test a subset from each class

16. Exercise Consider a factorial function, fact(n): if n < 0 or n >= 200, error if 0  n  20, exact value if 20 < n < 200, approximate value within 0.1%. Q: What equivalence classes can you see?

17. Simple Example Suppose you are building an airline reservation system. A traveler can be a child, an adult, or a senior. The price depends on the type of traveler. The seat reservation does not depend on the type of traveler. Q: How many test cases can you identify for the reservation component and the billing component?

18. Heuristicsfor Finding Equivalence Classes Identify restrictions for inputs and outputs in the specification.

19. Heuristicsfor Finding Equivalence Classes Identify restrictions for inputs and outputs in the specification. If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).

20. Heuristicsfor Finding Equivalence Classes Identify restrictions for inputs and outputs in the specification. If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN). If a number of values is required, create one valid and one or more invalid classes.

21. Heuristicsfor Finding Equivalence Classes Identify restrictions for inputs and outputs in the specification. If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN). If a number of values is required, create one valid and one or more invalid classes. If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.

22. Heuristicsfor Finding Equivalence Classes Identify restrictions for inputs and outputs in the specification. If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN). If a number of values is required, create one valid and one or more invalid classes. If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set. If there is a condition, create two classes, one satisfying and one not satisfying the condition.

23. Exercise Define test cases for a program that reserves a golf tee time. The standard green fee is $65 on weekdays (Monday-Friday) and $80 on weekend (Saturday and Sunday). However, an El Paso resident pays a reduced green fee of $45 and $60 on weekdays and weekend, respectively. A senior (of age 60+) pays only $40 and $50 on weekdays and weekend, respectively. A junior (of age <17) pays only $20 and $30 on weekdays and weekend, respectively. Q: How many equivalence classes? Define them. Q: Define one test case per equivalence class.

24. Outline Basic concepts Black box testing Basic concepts Equivalence classes Boundary values Decision tables … White box testing

25. Boundary Values Observations Programs that fail at interior elements of an equivalence class usually fail at the boundaries too. Programmers often make errors on boundary conditions (e.g., branch/loop condition x <= 10 instead of x < 10). Test the boundaries if it should work for 1-99, test 0, 1, 99, 100. if it works for A-Z, try @, A, Z, [, a, and z The hard part is identifying boundaries.

26. Hints If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11 It may be possible to test the boundaries of outputs, also. For ordered sets, check the first and last elements. For complex data structures, the empty list, full lists, the zero array, and the null pointer should be tested. Extremely large data sets should be tested. Check for off-by-one errors.

27. More Hints Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed) Numeric limits (e.g., test 255 and 256 for 8-bit values) Implementation limits (e.g., max array size)

28. Example Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz.

29. Example Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz. Voltage boundaries 189 volts (below the lowest boundary) 190 volts (lowest boundary) 240 volts (highest boundary) 241 volts (above the highest boundary) Cycles (hertz) boundaries 49 hertz (below) 50 hertz (lowest) 60 hertz (highest) 61 hertz (above)

30. Example Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz. Voltage boundaries 189 volts (below the lowest boundary) 190 volts (lowest boundary) 240 volts (highest boundary) 241 volts (above the highest boundary) Cycles (hertz) boundaries 49 hertz (below) 50 hertz (lowest) 60 hertz (highest) 61 hertz (above) How to combine? Exhaustive, Pairwise (later), One extremes with others fixed.

31. Exercise Determine the boundary values for US Postal Service ZIP codes (5 digits such as 79912 or 5 digits hyphen 4 digits such as 79912-1818). Determine the boundary values for a 15-character last name entry. Hints: Two kinds of boundaries---size boundary and value boundary.

32. Outline Basic concepts Black box testing Basic concepts Equivalence classes Boundary values Decision tables Pairwise testing State transitions Use cases White box testing

33. Decision Tables Construct a table (to help organize the testing) Identify each rule or condition in the system that depends on some input For each input to one of these rules, list the combinations of inputs and the expected results

34. Example Theater ticket prices are discounted for senior citizens and students.

35. Exercise Construct a decision table for a program that reserves a golf tee time. The standard green fee is $65 on weekdays (Monday-Friday) and $80 on weekend (Saturday and Sunday). However, an El Paso resident pays a reduced green fee of $45 and $60 on weekdays and weekend, respectively. A senior (of age 60+) pays only $40 and $50 on weekdays and weekend, respectively. A junior (of age <17) pays only $20 and $30 on weekdays and weekend, respectively.

36. Consider a website that must: operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome. work using RealPlayer, MediaPlayer, or no plug-in. run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0 accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers. How many different configurations are there? Pairwise Testing: Motivation

37. Consider a website that must: operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome. work using RealPlayer, MediaPlayer, or no plug-in. run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0 accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers. How many different configurations are there? A: 9 browsers x 3 plug-ins x 7 OS’s x 3 web servers x 3 server OS’s = 1701 combinations Motivation

38. Many such examples (i.e., finite sets of discrete values). A bank is ready to test a data processing system Customer types: Gold, Platinum, Business, Non profits Account types: Checking, Savings, Mortgages, Consumer loans, Commercial loans States (with different rules): CA, NV, UT, ID, AZ, NM How many different configurations are there? Motivation

39. Approaches? When given a large number of combinations: • Give up and don’t test • Test all combinations: miss targets, delay product launch, and go out of business • Choose one or two cases • Choose a few that the programmers already ran • Choose the tests that are easy to create • List all combinations and choose first few • List all combinations and randomly choose some • “Magically” choose a small subset with high probability of revealing defects

40. Pairwise Testing • Combinatorial testing technique in which every pair of input parameters of software is tested. • Reasonable cost-benefit compromise • Much faster than exhaustive testing • More effective than less exhaustive methods that fail to exercise all possible pairs of input parameters • Why? Majority of software failures are caused by a single input parameter or a combination of two input parameters. • Each pair of input parameter values should be captured at least by one test case. • However, finding the least number of test cases is an NP-complete problem.

41. Example • Consider software that takes three input parameters, say x, y, and z. • If each parameter can have three different values, then there will be 27 different pairs: (x1, y1), (x1, y2), …, (y3, z3). • A test case (x1, y3, z2), for example, captures three of these 27 pairs: (x1, y3), (x1, z2), and (y3, z3). • By selecting test cases judiciously, all pairs of input parameters can be exercised with a minimum number of test cases; e.g., a set of 9 test cases can capture all 27 pairs of three parameters, each with three different values.

42. State Transitions Use state transitions (e.g., UML state machine diagrams) to derive test inputs Cover a state transition diagram, e.g., Visit each state at least once Trigger each event at least once Exercise each transition at least once Exercise each path at least once* * Might not be possible.

43. Example and Exercise Customer makes a reservation and has a limited time to pay. May cancel at any time. printed Paid paid Ticketed ordered/ start timer Reserved delivered canceled canceled/ refund canceled/ refund timer expired Used Cancelled Unpaid Groups: How many tests to visit each state, trigger each event, exercise each transition, and exercise each path?

44. Use Cases Use the use cases to specify test cases Use case specifies both normal, alternate, and exceptional operations However, use cases may not have sufficient details, esp. for unit testing

45. Outline Basic concepts Black box testing White box testing Control flow graph (CFG) Statement coverage …

46. White Box Testing Test set is derived from structure of (source) code Also known as: Glass box testing Structural testing Often use a graph called a control flow graph (CFG) To represent code structure To cover it (CFG), e.g., all statements Input Output

47. Control Flow Graphs Programs are made of three kinds of statements: Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements)

48. Control Flow Graphs Programs are made of three kinds of statements: Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements) CFG: visual representation of flow of control Node represents a sequence of statements with single entry and single exit Edge represents transfer of control from one node to another

49. Control Flow Graph (CFG) Sequence If-then-else Iterative If-then

50. Control Flow Graph (CFG) Join Join Sequence If-then-else Iterative If-then When drawing CFG, ensure that there is one exit: include the join node if needed