Random Testing

1. Random Testing Tor Stålhane Jonas G. Brustad

2. What is random testing The principle of random testing is simple and can be described as follows: • For each input parameter, generate a random but legal value. • Apply the full set of inputs to the SUT • Register the result and go back to step 1.

3. Chen’s observation Inputs that are close to each others in the input domain tends to go through the same path. Thus, in order to find most of the errors, we should spread the test cases as much as possible. This approach is called Adaptive Random Testing. We will look at four approaches: • Partition Adaptive Random Testing • Basic Random Testing – RT • Basic Adaptive Random Testing – ART • Mirror Adaptive Random Testing – MART

4. Block failure pattern

5. Strip failure pattern

6. Point failure pattern

7. Some notation • D: input domain size • n: number of test cases • m: number of tests that fail • F = D/m. Note that • Large F => small m => few errors detected • Small F => large m => many errors detected • q = 1/F – failure rate • Frel = Fobs * q

8. Partition Adaptive Random Testing Tor Stålhane Jonas G. Brustad

9. ART by random partitioning – 1 Algorithm for a two-dimensional case • Start with C = {(Xmin, Ymin), (Xmax, Ymax)} • Draw a random point in C = (X1, Y1). This will split C into four regions – R1, R2, R3 and R4. Select T = max area {R1, R2, R3, R4}See next slide.

10. ART by random partitioning – 2 (Xmax, Ymax) Select a test (X1, Y1) in T. If it is a failure, report the failure and stop. Otherwise, split T in the same way as we split C – see nextslide R1 T = R2 (X1, Y1) R3 R4 {(Xmin, Ymin),

11. ART by random partitioning – 3 (Xmax, Ymax) (X2, Y2) Select a test (X2, Y2) in T. If it is a failure, report the failure and stop.Otherwise, repeatthe process R1 T (X1, Y1) R3 R4 {(Xmin, Ymin),

12. ART by bisection – 1 Algorithm for a two-dimensional case • Start with C = {(Xmin, Ymin), (Xmax, Ymax)} • Draw a random test in C – (X1, Y1). If it fails we are finished. Otherwise split C in two equal parts – see next slide.

13. ART by bisection – 2 (Xmax, Ymax) Select a test (X2, Y2) in the untested half of C. If it is a failure, report the failure and stop. Otherwise, split C again – see nextslide (X2, Y2) (X1, Y1) {(Xmin, Ymin),

14. ART by bisection – 3 (Xmax, Ymax) Select a test (X2, Y2) in T. If it is a failure report, the failure and stop.Otherwise, repeatthe process for eachpart that we havenot tested – (X3, Y3) and (X4, Y4) (X2, Y2) (X3, Y3) (X1, Y1) (X4, Y4) {(Xmin, Ymin),

15. The exclusion factor – 1 All types of Adaptive Random Testing (ART) can be improved by introducing the exclusion factor, usually denoted by f. This factor will force the new tests away from the tests that have already been run. The optimal factor value will vary, depending on the failure rate and on the failure pattern – block, strip or point.

16. The exclusion factor – 2

17. The exclusion factor – 3 Based on this, we have chosen f = 0.4 as the best value.

18. Comparisons

19. Basic Adaptive Random Testing Tor Stålhane Jonas G. Brustad

20. Fixed Size Candidate Set - FSCS

21. Max distance – 1 Let a and b be two n-dimensional inputs {a1, a2, ..., an} and {b1, b2, ..., bn}. E.g. in a two-dimensional space we have the twoparameters a ={1, 2} and b = {2, 5}. Then we havedist(a, b) = sqrt(1 + 9) = 3.16

22. Max distance – 2 Let T and C be two disjoint sets T = {t1, t2, ...tn} is the set of executed tests C = {c1, c2, ...ck} is the candidate set. Find the ch that satisfies: This criterion will spread the test cases evenly by finding the largest minimum distance betweenthe next test case – selected from C – and the already executed test cases in T.

23. Max distance algorithm

24. A small example – 1 We have two data sets: • T = {(1, 1), (3,4)} – already executed tests • C = {(1, 2), (3, 1)} – candidate test set t2 4 3 c1 2 1 c2 t1 2 1 3

25. A small example – 2 Using the max distance algorithm we get: • j = 1 => (c1, t1) dist= 1.0, (c1, t2) dist = 2.8 • j = 2 => (c2, t1) dist= 2.0, (c2, t2) dist= 3.0 • min(dist) = 1.0 and the first distance larger than min(dist) is 2.0 => Next test is c2 =(3, 1)

26. Test comparison – 1 Defect types seeded: • AOR: Arithmetic Operator Replacement • ROR: Relational Operator Replacement • SVR: Scalar Variable Replacement • CR: Constant Replacement

27. Results with RT 1

28. Results with ART (a) and FSCS (r) N

29. Test comparison – 2

30. Mirror Adaptive Random Testing Tor Stålhane Jonas G. Brustad

31. The problem with ART All versions of ART require a large amount of computations due to the distance calculations and comparisons. The MART – Mirror ART – is simpler and requires less computation.

32. The MART procedure The procedure has four steps: • Partition the input domain into m disjoint subdomains. One is chosen as source subdomain. The rest are mirror subdomains • Apply the D-ART process to generate the next test case from the source subdomain. Execute this test case and quit if we find a defect. • Apply the mirror function to the test case from step 2 to generate a test case for each mirror subdomain. Execute the test cases in sequential order and stop when we find a defect. • Repeat steps 2 and 3 until finding the first failure or until reaching the stopping condition.

33. Mirror partitioning Below we see several ways to create mirror partitions: X2Y1 => X is bisected, y is unchanged X2Y2 => both X and Y are bisected X4Y2 => X is split into four parts, Y is bisected X4Y1 => X is split into four parts, Y is unchanged

34. The D-ART process • Set E to be the empty set • Select a random test case from the input domain and execute it. If no failure, add the test case to E, otherwise stop. • Construct C = {c1, c2,…, ck}, where all ci are randomly selected and E and C are disjoint. • Let n = |E| and select cj so that • Repeat steps 3 and 4 until first defect is found