1 / 30

Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing

Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing. Raghu Kacker, NIST Yu Lei, UTA 5/15/06. Outline. Introduction The IPO Strategy FireEye: An N-Way Testing Tool Related Work Conclusion. Why Software Testing?.

abiola
Download Presentation

Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Combinatorial Testing Using Covering Arrays- Going Beyond Pairwise Testing Raghu Kacker, NIST Yu Lei, UTA 5/15/06

  2. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 2

  3. Why Software Testing? • Modern society is increasingly dependent on the quality of software systems. • A NIST study reports that software failure costs the US economy billions of dollars every year. • Testing (or dynamic analysis) is the most widely used approach to ensuring software quality • Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale Combinatorial Testing Using Covering Arrays 3

  4. Testing Process • The testing process consists of three stages: • Test Generation: Generate test data • For model-based testing, a model (or abstraction) of the system has to be built. • Test Execution: Test setup and the actual test runs • Test Evaluation: Check if the output is in line with expectations Combinatorial Testing Using Covering Arrays 4

  5. All About Trade-Off • Testing is labor intensive and can be very costly • estimated to often consume more than 50% of the development cost • Exhaustivetesting is often impractical due to resource constraints • From a certain perspective, testing is basically about making a good trade-off between testeffort and quality assurance. Combinatorial Testing Using Covering Arrays 5

  6. Combinatorial Testing • Generates tests from an input parameter model by combining the values of the parameters. • Requires lightweight specification and no knowledge about the implementation structure • Can be virtually applied to any software and at different levels of abstraction • Can be implemented as a push-button feature • Usually performed to achieve t-way coverage, i.e. guarantees to cover every t-way interaction • Motivation: Not every parameter contributes to every fault • Can dramatically reduce the number of tests while still preserving important fault detection capabilities. Combinatorial Testing Using Covering Arrays 6

  7. The ASTUCA Project • Develop new methods and tools for efficient t-way testing for up to 6-way testing • Create new algorithms for efficient test set construction • Provide adequate support for parameter relations and constraints • Explore integration with other software testing tools • Conduct empirical evaluation of t-way testing in an industrial setting • A collaborative effort among the following institutions: • The US National Institute of Standards and Technology • George Mason University • The University of Texas at Arlington Combinatorial Testing Using Covering Arrays 7

  8. State-of-the-Art • Existing work has mainly focused on pairwise testing • Many failures are caused by the interaction involving more than two parameters • For certain software, pairwise testing discovers a relatively low percentage of faults • e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults. • Increased coverage leads to a higher level of assurance • Many applications, e.g., security protocols, have strict requirements on test coverage Combinatorial Testing Using Covering Arrays 8

  9. Technical Challenges • The computational complexity for t-way testing grows rapidly as the value of t increases • New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets • The number of tests also grows rapidly as the value of t increases • Impractical to manually execute and inspect the results of a large number of test runs • Test generation, test execution, and test evaluation must integrate together to enable test automation Combinatorial Testing Using Covering Arrays 9

  10. Terminology • N-Way Test set -> N-Way Covering array • Tests -> Rows • Parameters -> Factors or Columns • Values -> Levels Combinatorial Testing Using Covering Arrays 10

  11. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 11

  12. The Framework (1) • Builds a t-way test set in an incremental manner • A t-way test set is first constructed for the first t parameters, • Then, the test set is extended to generate a t-way test set for the first t + 1 parameters • The test set is repeatedly extended for each additional parameter. • Two steps involved in each extension for a new parameter: • Horizontal growth: extends each existing test by adding one value of the new parameter • Vertical growth: adds new tests, if necessary Combinatorial Testing Using Covering Arrays 12

  13. The Framework (2) • Strategy In-Parameter-Order • begin • /* for the first t parameters p1, p2 , …, pt*/ • T := {(v1, v2, …, vt) | v1, v2, …, vt are values of p1, p2, …, Pk, respectively} • if n = t then stop; • /* for the remaining parameters */ • for parameter pi, i = t + 1, …, n do • begin • /* horizontal growth */ • for each test (v1, v2, …, vi-1) in T do • replace it with (v1, v2, …, vi-1, vi), where vi is a value of pi • /* vertical growth */ • while T does not cover all the interactions between pi and • each of p1, p2, …, pi-1do • add a new test for p1, p2, …, pito T; • end • end Combinatorial Testing Using Covering Arrays 13

  14. Example (1) • Consider a system with the following parameters and values: • parameter A has values A1 and A2 • parameter B has values B1 and B2, and • parameter C has values C1, C2, and C3 Combinatorial Testing Using Covering Arrays 14

  15. A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 A2 B1 C2 A1 B2 C3 A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 Example (2) A B A1 B1 A1 B2 A2 B1 A2 B2 Horizontal Growth Vertical Growth Combinatorial Testing Using Covering Arrays 15

  16. Comparison to AETG (1) • A commercial tool developed by Telcordia, and protected by a US patent • Starts with an empty set and adds one (complete) test at a time • Each test is locally optimized to cover the most number of missing pairs: • Generate a random order of the parameters • Use a greedy algorithm to construct a test that covers the most uncovered pairs • Repeat the above two steps for a given number of times (suggested 50), and select the best one Combinatorial Testing Using Covering Arrays 16

  17. Comparison to AETG (2) A B C A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 A2 B1 C2 A1 B2 C3 A B C A1 B1 C1 A1 B2 C2 A B C A1 B1 C1 Adds the 1st test Adds the 2nd test Adds the last test Combinatorial Testing Using Covering Arrays 17

  18. IPO vs AETG • IPO is deterministic, whereasAETG is inherently non-deterministic • IPO has a lower order of complexity, both in terms of time and space, than AETG • IPO constructs a test set one parameter at a time and in a more incremental nature. • The results generated by IPO are still competitive to those generated by AETG. • IPO is more flexible than AETG • IPO can take a test set for a subsystem (i.e., for a subset of parameters) and then extend it to a complete set for the entire system Combinatorial Testing Using Covering Arrays 18

  19. Outline • Introduction • The IPO Strategy • FireEye: A Prototype Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 19

  20. Major Features • Uses Java as the programming language • Relatively easier to program, leading to reduced development time and ease of maintenance • Supports the concept of “write-once-run-everywhere” • Data structures are carefully designed to optimize the runtime performance • A hierarchical structure is used to manage the possible interactions • Allows an incomplete test set to be extended to a complete one • Add new tests or parameters, if necessary, to achieve t-way coverage Combinatorial Testing Using Covering Arrays 20

  21. Initial Results (1) • [System] • Name: Test Configuration for TCAS • [Parameter] • -- only compare with MINSEP and MAXALTDIFF • Cur_Vertical_Sep : 299, 300, 601 • High_Confidence : TRUE, FALSE • Two_of_Three_Reports_Valid : TRUE, FALSE • -- Low and High, only compare with Other_Tracked_Alt • Own_Tracked_Alt : 1, 2 • Other_Tracked_Alt : 1, 2 • -- only compare with OLEV • Own_Tracked_Alt_Rate : 600, 601 • Alt_Layer_Value : 0, 1, 2, 3 • -- compare with each other (also see NOZCROSS) and with ALIM • Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 • Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 • Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND • Other_Capability : TCAS_TA, OTHER • Climb_Inhibit : TRUE, FALSE Combinatorial Testing Using Covering Arrays 21

  22. Initial Results (2) All the experiments are performed on a desktop with 1.2GHZ CPU and 1GB memory. Combinatorial Testing Using Covering Arrays 22

  23. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 23

  24. Classification • Search-Based methods that are mainly developed by computer scientists • AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), PairTest • Algebraic methods that are mainly developed by mathematicians • Orthogonal Arrays • Recursive Construction Combinatorial Testing Using Covering Arrays 24

  25. Orthogonal Arrays • Orthogonal arrays can be constructed very fast and are always optimal • Any extra test will cause a pair to be covered for more than once • However, there are several limitations: • Orthogonal arrays do not always exist • Every parameter must have the same number v of values • Existing methods often require v be a prime power. • Every t-way interaction must be covered at the same number of times Combinatorial Testing Using Covering Arrays 25

  26. Recursive Construction • Covering arrays are a more general structure, which requires every t-way interaction be covered at least once • Constructing a covering array from one or more covering arrays with smaller parameter sets • Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes Combinatorial Testing Using Covering Arrays 26

  27. Search-Based vs Algebraic Methods • Search-based methods: • Advantages: no restrictions on the input model, and very flexible, e.g., relatively easier to support parameter relations and constraints • Disadvantages: explicit search takes time, the resulting test sets are not optimal • Algebraic methods: • Advantages: very fast, and often produces optimal results • Disadvantages: limited applicability, difficult to support parameter relations and constraints • The advantages and disadvantages of the two types of methods seem to complement with each other Combinatorial Testing Using Covering Arrays 27

  28. Outline • Introduction • The IPO Strategy • FireEye: An N-Way Testing Tool • Related Work • Conclusion Combinatorial Testing Using Covering Arrays 28

  29. Conclusion • Combinatorial testing is a well-defined problem and has been used widely in practice. • The IPO strategy has a lower order of complexity than AETG, and still produces competitive results. • Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible. • Going beyond 2-way testing presents challenges and opportunities to the area of combinatorial testing. Combinatorial Testing Using Covering Arrays 29

  30. References • Boroday S. Y. and Grunskii I. S., “Recursive generation of locally complete tests,” Cybernetics and Systems Analysis 28 (1992), 20-25. • K. A., Bush, “Orthogonal arrays of index unity,” Annals of Mathematical Statistics, 23 (1952), 426-434. • D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7. • M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing,” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland. • R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6. • Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays,” Discrete Mathematics 284(1-3): 149-156 (2004) • Y. Lei and K. C. Tai , “In-parameter-order: a test generation strategy for pairwise testing,” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261. • K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1. Combinatorial Testing Using Covering Arrays 30

More Related