Testing and Test Case Development

1. Testing and Test Case Development • A “primitive” method of testing, with NO test preparation, may include the following steps : • Initiate the system • Provide the requested input(s) {unplanned value} • Observe the outputs/results • Record and analyze the results • Note that such testing (ad hoc or random) is very difficult to analyze: • Is the test coverage complete? • How much duplication exist?

2. Checklist Testing • A more orderly testing than ad hoc testing is to question the “coverage” in some systematic way. • A Check-listof coverage is developed: • The test cases designed & developed are based on the checklist items which maycover different items such as: • Major Functionalities (black-box) • Modules in the system (white-box) • All Error Messages or All UI Screens (white-box) • The test execution may be terminated & analyzed via: • the coverage - the % of the checklist test cases executed • the coverage - the % of checklist items executed successfully

3. Checklist Example • An application “functional” checklist (perhaps derived from Requirements doc.) of an on-line-order software may include the following: • Entering the order items • Validating the order items • Checking availability/price of order items • Summarizing the order items and the total price into an invoice • Display and validate the invoice You may “check-off” the items as you tested each of the above functionalities.

4. More on Checklist • A checklist may be further expanded to form a “tree” of checklists to provide some depth. • Several checklists may be combined to form a “matrix” of checklists. Some logical intersection of checklists may be developed into a “new” checklist. • Some drawbacks of checklist: • Too high level to cover some of the details • One high level item may require extensive analysis (complex scenario and thus require several related test cases to test a scenario.) • There may exist “overlaps” at high level • Complex interactions may also exist besides overlapping • Too difficult to combine multiple views such as black-box and white-box testing even with combined checklists. Can we improve on checklist ? ----

5. Partition • We care about partition because we want to divide our test cases up so that they (a)uniquely(with non or minimal duplication) cover(b) all the situations of interest. • The formal mathematical definition of a partition is: • Consider a set S • The set S is partitioned into subsets C1, C2, ----, Cn if: • For any x, y, x≠y then Cx ∩ Cy = Ø (there is no overlap between Ci’s) • ∑ Ci = S (union of all Ci’s is S)

6. Partition of set S of marbles “by size” C1 C5 C3 C2 C4 • Note that: 1) For any x, y, x≠y then Cx ∩ Cy = Ø (there is no overlap between Ci’s) • 2) ∑ Ci = S (union of all Ci’s is S)

7. Some More “Math” on Partition • Each partition, Cx, of set S is considered an equivalence class. (this is because every member inside Cx is the “same” with respect to the property, size.) A relation “= size” used to partition the set S. • Another example: the set S may be a set of dogs, and we partitioned S by the color of the dogs {black, white, brown, buff, gray, multi-color}. Then every dog in the partition “black” is the “same” with respect to that color. • The relation for members in an equivalence class satisfy the following rules. Let R represent the “equivalence” relation then: • R(a,b) => R(b,a) [symmetric] {e.g.} Samecolor(a,b) => Samecolor(b,a) • R(a,b) Λ R(b,c) => R(a,c) [transitive] {e.g.} { Samecolor(a.b) Λa Samecolor (b,c) } => Samecolor(a,c) • R(a,a) [reflexive] {e.g.} Samecolor (a,a)

8. Example of Testing Program Q • Consider a program Q that i) reads inputs a, b, c and ii) determines the roots, r1 and r2, for the quadratic equation ax2 +bx +c = 0. • From high school math, we remember the 2 roots for this equation is given by: r1 = [ -b + √(b2 -4ac) ]/2a and r2 = [ -b - √(b2 -4ac) ]/2a • To test this program , we may consider all kinds of approaches • How should we decide on the test cases for our testing of this program?

9. 1st Example of Test cases for the Program Q • Consider the inputs to the program Q: a, b, c • We could divide the three inputs into valid (numerical) and invalid (non-numerical) to see if program Q handles input validation properly • Then we would have 2 cases for each input variable or 2 x 2 x 2 = 8 test cases to cover all the possible “valid-invalid” input combinations: • (a = valid, b= valid, c= valid) • (a = valid, b= valid , c= invalid) • (a = valid, b= invalid, c= valid) • (a= valid, b= invalid, c= invalid) • (a= invalid, b= valid, c= valid) • (a= invalid, b= valid, c= invalid) • (a= invalid, b= invalid, c= valid) • (a= invalid, b= invalid, c= invalid) • Note: that this is a checklist and is also a partitioning of the input based on our coming up with the relation of “valid and invalid’ inputs

10. Partitioning Input, (a,b,c), into 8 Equivalence Categories • (a = valid, b= valid, • c= valid) • (a = invalid, b= valid, • c= valid) • (a = invalid, b= valid, • c= invalid) • (a = valid, b= valid, • c= invalid) • (a = invalid, b= invalid, • c= valid) • (a = valid, b= invalid, • c= valid) • (a = valid, b= invalid, • c= invalid) • (a = invalid, b= invalid, • c= invalid) • Note that within each equivalence category there may be infinite choices! • Note that Choosing 1 from each category covers a) all 8 categories and • b) each category is unique ---- partition

11. 2nd Example of Test cases for the Program Q • Again, consider the inputs to the program Q: a, b, c • We consider the situation of =0, >0 and <0 values for each input to test the implementation of algorithm for root • Then we will have 3 x 3 x 3 = 27 test cases to cover all the possible combinations: (a=0, b=0, c =0) (a>0, b=0, c =0) (a<0, b=0, c =0) (a=0, b=0, c >0) (a>0, b=0, c >0) (a<0, b=0, c >0) (a=0, b=0, c <0) (a>0, b=0, c <0) (a<0, b=0, c <0) (a=0, b>0, c =0) (a>0, b>0, c =0) (a<0, b>0, c =0) (a=0, b>0, c >0) (a>0, b>0, c >0) (a<0, b>0, c >0) (a=0, b>0, c <0) (a>0, b>0, c <0) (a<0, b>0, c <0) (a=0, b<0, c =0) (a>0, b<0, c =0) (a<0, b<0, c =0) (a=0, b<0, c >0) (a>0, b<0, c >0) (a<0, b<0, c >0) (a=0, b<0, c <0) (a>0, b<0, c <0) (a<0, b<0, c <0) • Note that this is also a checklist and is also a partitioning of the input based on our coming up with the relation of =0, >0, and <0 values for the 3 inputs

12. 3rd Example of Test cases for the Program Q • This time consider the outputs to the program Q: r1, r2 • We test the two major categories of possible outputs: a) r1 = r2 and b) r1 ≠ r2 • If (b2 -4ac) = 0, then r1 = r2 because r1 = (-b + 0)/2a and r2 = (-b – 0)/2a • If (b2 -4ac) >0, then r1 ≠ r2 • We have partitioned our “outputs” into 2 partitions : r1=r2 and r1 ≠ r2 ; this in turn partitioned our inputs of a,b,c into: • set of (a,b,c)’s such that b2 = 4ac • set of (a,b,c)’s such that b2 > 4ac 1)Note that this is a checklist and also a partition of 2 sets of input (a,b,c)’s based on covering the 2 partitions of outputs where r1 = r2 and r1 ≠ r2. 2)Note that your text includes the case of output being imaginary number.

13. Partitions • Test Case partitioning technique may be applied over different types of properties of interest: • Inputs • Outputs • Functionalities • Usages • Logical paths • etc. • We can combine the different partitioning (Examples 1, 2, and 3) to cover different aspect of Program Q: input validation, algorithm correctness, output coverage)

14. Usage Based Testing • Usage Based testing may be viewed as a type of partitioning. • We formulate a set of user-operational profiles (partitioning the users by some characteristics: e.g. user role & access of different functionalities by role) • We may attach probabilities to the user profile (e.g. different probabilities of access to different functionalities based on the role.) • We may formulate a resource usage profile (partitioning the resources by some characteristics: e.g. number of times accessed by resources) • We may attach frequency of usage count to each resource (e.g. develop a statistical usage frequency distribution of the resources) The idea is to test more on the more heavily “used” categories because more heavily used category is more “important”

15. User-Operational Profile Example(naïve user versus experienced user) (20/20) (20/20) prob = .25 √ term naïve users (20/80) default f1 f4 (5/5) prob = .063 (5/10) f4 term Start (80 users) (5/10) (5/5) prob = .063 (10/60) f1 f6 term experienced users (60/80) (40/40) (40/40) (40/60) prob = .50√√ term f7 f2 Choice of f1,f2,f3 (10/60) (3/10) (3/3) prob = .037 term f4 f3 (7/10) (7/7) prob = .087 Note that 2 usage cases take up 75% of operational profile ! f5 term

16. Some Interesting Observations/Usageof the User Operational Profile • From a testing priority and resource allocation perspective, we may conduct our testing in a sequence equal to the highest probability scenario (choice of f1,f2,f3 - f2 – f7 ) to the lowest probability scenario (choice of f1,f2,f3 – f3 – f4). • If we want to claim some form of “coverage,” we may use the user-profile and test all the scenarios thoroughly except (choice of f1,f2,f3 – f3 – f4) and claim 96% coverage reliability.

17. Resource-Usage Profile Example(DB-table Usage Frequency distribution) Usage/Month 9k times employeetable is “used” the most; thus more test cases should be created to for accessing that table 6k times 3ktimes employee payroll benefits dependent tax tax DB - Tables