Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing

1. Combinatorial Testing Using Covering Arrays- Going Beyond Pairwise Testing Raghu Kacker, NIST Yu Lei, UTA 5/15/06

2. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 2

3. Why Software Testing? • Modern society is increasingly dependent on the quality of software systems. • A NIST study reports that software failure costs the US economy billions of dollars every year. • Testing (or dynamic analysis) is the most widely used approach to ensuring software quality • Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale Combinatorial Testing Using Covering Arrays 3

4. Testing Process • The testing process consists of three stages: • Test Generation: Generate test data • For model-based testing, a model (or abstraction) of the system has to be built. • Test Execution: Test setup and the actual test runs • Test Evaluation: Check if the output is in line with expectations Combinatorial Testing Using Covering Arrays 4

5. All About Trade-Off • Testing is labor intensive and can be very costly • estimated to often consume more than 50% of the development cost • Exhaustivetesting is often impractical due to resource constraints • From a certain perspective, testing is basically about making a good trade-off between testeffort and quality assurance. Combinatorial Testing Using Covering Arrays 5

6. Combinatorial Testing • Generates tests from an input parameter model by combining the values of the parameters. • Requires lightweight specification and no knowledge about the implementation structure • Can be virtually applied to any software and at different levels of abstraction • Can be implemented as a push-button feature • Usually performed to achieve t-way coverage, i.e. guarantees to cover every t-way interaction • Motivation: Not every parameter contributes to every fault • Can dramatically reduce the number of tests while still preserving important fault detection capabilities. Combinatorial Testing Using Covering Arrays 6

7. The ASTUCA Project • Develop new methods and tools for efficient t-way testing for up to 6-way testing • Create new algorithms for efficient test set construction • Provide adequate support for parameter relations and constraints • Explore integration with other software testing tools • Conduct empirical evaluation of t-way testing in an industrial setting • A collaborative effort among the following institutions: • The US National Institute of Standards and Technology • George Mason University • The University of Texas at Arlington Combinatorial Testing Using Covering Arrays 7

8. State-of-the-Art • Existing work has mainly focused on pairwise testing • Many failures are caused by the interaction involving more than two parameters • For certain software, pairwise testing discovers a relatively low percentage of faults • e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults. • Increased coverage leads to a higher level of assurance • Many applications, e.g., security protocols, have strict requirements on test coverage Combinatorial Testing Using Covering Arrays 8

9. Technical Challenges • The computational complexity for t-way testing grows rapidly as the value of t increases • New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets • The number of tests also grows rapidly as the value of t increases • Impractical to manually execute and inspect the results of a large number of test runs • Test generation, test execution, and test evaluation must integrate together to enable test automation Combinatorial Testing Using Covering Arrays 9

10. Terminology • N-Way Test set -> N-Way Covering array • Tests -> Rows • Parameters -> Factors or Columns • Values -> Levels Combinatorial Testing Using Covering Arrays 10

11. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 11

12. The Framework (1) • Builds a t-way test set in an incremental manner • A t-way test set is first constructed for the first t parameters, • Then, the test set is extended to generate a t-way test set for the first t + 1 parameters • The test set is repeatedly extended for each additional parameter. • Two steps involved in each extension for a new parameter: • Horizontal growth: extends each existing test by adding one value of the new parameter • Vertical growth: adds new tests, if necessary Combinatorial Testing Using Covering Arrays 12

13. The Framework (2) • Strategy In-Parameter-Order • begin • /* for the first t parameters p1, p2 , …, pt*/ • T := {(v1, v2, …, vt) | v1, v2, …, vt are values of p1, p2, …, Pk, respectively} • if n = t then stop; • /* for the remaining parameters */ • for parameter pi, i = t + 1, …, n do • begin • /* horizontal growth */ • for each test (v1, v2, …, vi-1) in T do • replace it with (v1, v2, …, vi-1, vi), where vi is a value of pi • /* vertical growth */ • while T does not cover all the interactions between pi and • each of p1, p2, …, pi-1do • add a new test for p1, p2, …, pito T; • end • end Combinatorial Testing Using Covering Arrays 13

14. Example (1) • Consider a system with the following parameters and values: • parameter A has values A1 and A2 • parameter B has values B1 and B2, and • parameter C has values C1, C2, and C3 Combinatorial Testing Using Covering Arrays 14

15. A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 A2 B1 C2 A1 B2 C3 A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 Example (2) A B A1 B1 A1 B2 A2 B1 A2 B2 Horizontal Growth Vertical Growth Combinatorial Testing Using Covering Arrays 15

16. Comparison to AETG (1) • A commercial tool developed by Telcordia, and protected by a US patent • Starts with an empty set and adds one (complete) test at a time • Each test is locally optimized to cover the most number of missing pairs: • Generate a random order of the parameters • Use a greedy algorithm to construct a test that covers the most uncovered pairs • Repeat the above two steps for a given number of times (suggested 50), and select the best one Combinatorial Testing Using Covering Arrays 16

17. Comparison to AETG (2) A B C A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 A2 B1 C2 A1 B2 C3 A B C A1 B1 C1 A1 B2 C2 A B C A1 B1 C1 Adds the 1st test Adds the 2nd test Adds the last test Combinatorial Testing Using Covering Arrays 17

18. IPO vs AETG • IPO is deterministic, whereasAETG is inherently non-deterministic • IPO has a lower order of complexity, both in terms of time and space, than AETG • IPO constructs a test set one parameter at a time and in a more incremental nature. • The results generated by IPO are still competitive to those generated by AETG. • IPO is more flexible than AETG • IPO can take a test set for a subsystem (i.e., for a subset of parameters) and then extend it to a complete set for the entire system Combinatorial Testing Using Covering Arrays 18

19. Outline • Introduction • The IPO Strategy • FireEye: A Prototype Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 19

20. Major Features • Uses Java as the programming language • Relatively easier to program, leading to reduced development time and ease of maintenance • Supports the concept of “write-once-run-everywhere” • Data structures are carefully designed to optimize the runtime performance • A hierarchical structure is used to manage the possible interactions • Allows an incomplete test set to be extended to a complete one • Add new tests or parameters, if necessary, to achieve t-way coverage Combinatorial Testing Using Covering Arrays 20

21. Initial Results (1) • [System] • Name: Test Configuration for TCAS • [Parameter] • -- only compare with MINSEP and MAXALTDIFF • Cur_Vertical_Sep : 299, 300, 601 • High_Confidence : TRUE, FALSE • Two_of_Three_Reports_Valid : TRUE, FALSE • -- Low and High, only compare with Other_Tracked_Alt • Own_Tracked_Alt : 1, 2 • Other_Tracked_Alt : 1, 2 • -- only compare with OLEV • Own_Tracked_Alt_Rate : 600, 601 • Alt_Layer_Value : 0, 1, 2, 3 • -- compare with each other (also see NOZCROSS) and with ALIM • Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 • Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 • Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND • Other_Capability : TCAS_TA, OTHER • Climb_Inhibit : TRUE, FALSE Combinatorial Testing Using Covering Arrays 21

22. Initial Results (2) All the experiments are performed on a desktop with 1.2GHZ CPU and 1GB memory. Combinatorial Testing Using Covering Arrays 22

23. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 23

24. Classification • Search-Based methods that are mainly developed by computer scientists • AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), PairTest • Algebraic methods that are mainly developed by mathematicians • Orthogonal Arrays • Recursive Construction Combinatorial Testing Using Covering Arrays 24

25. Orthogonal Arrays • Orthogonal arrays can be constructed very fast and are always optimal • Any extra test will cause a pair to be covered for more than once • However, there are several limitations: • Orthogonal arrays do not always exist • Every parameter must have the same number v of values • Existing methods often require v be a prime power. • Every t-way interaction must be covered at the same number of times Combinatorial Testing Using Covering Arrays 25

26. Recursive Construction • Covering arrays are a more general structure, which requires every t-way interaction be covered at least once • Constructing a covering array from one or more covering arrays with smaller parameter sets • Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes Combinatorial Testing Using Covering Arrays 26

27. Search-Based vs Algebraic Methods • Search-based methods: • Advantages: no restrictions on the input model, and very flexible, e.g., relatively easier to support parameter relations and constraints • Disadvantages: explicit search takes time, the resulting test sets are not optimal • Algebraic methods: • Advantages: very fast, and often produces optimal results • Disadvantages: limited applicability, difficult to support parameter relations and constraints • The advantages and disadvantages of the two types of methods seem to complement with each other Combinatorial Testing Using Covering Arrays 27

28. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 28

29. Conclusion • Combinatorial testing is a well-defined problem and has been used widely in practice. • The IPO strategy has a lower order of complexity than AETG, and still produces competitive results. • Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible. • Going beyond 2-way testing presents challenges and opportunities to the area of combinatorial testing. Combinatorial Testing Using Covering Arrays 29

30. References • Boroday S. Y. and Grunskii I. S., “Recursive generation of locally complete tests,” Cybernetics and Systems Analysis 28 (1992), 20-25. • K. A., Bush, “Orthogonal arrays of index unity,” Annals of Mathematical Statistics, 23 (1952), 426-434. • D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7. • M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing,” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland. • R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6. • Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays,” Discrete Mathematics 284(1-3): 149-156 (2004) • Y. Lei and K. C. Tai , “In-parameter-order: a test generation strategy for pairwise testing,” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261. • K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1. Combinatorial Testing Using Covering Arrays 30