1 / 26

Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing. Andrea Calvagna – University of Catania - IT Angelo Gargantini – University of Bergamo – IT TAP 2009 – Zürich – 2nd July 2009. Outline.

rigg
Download Presentation

Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Combining Satisfiability Solving and Heuristics toConstrained Combinatorial Interaction Testing Andrea Calvagna – Universityof Catania - IT Angelo Gargantini – Universityof Bergamo – IT TAP 2009 – Zürich – 2nd July 2009

  2. Outline • Background on Combinatorial Interaction Testing • constraints • logic approach to CIT [TAP 08] • Test generation by satisfiability solving in the presence of constraints – by using Yices • Heuristics - for ordering test predicates • Experimental assessment Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  3. Interaction testing • IT : interaction testing • identify independent features and representative values (the inputs) • test the interaction between them • highly configurable systems • Software for families of products (like cell phones) • Sw like gcc compiler > 1400 optional features • … Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  4. How much to test? • 34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests? • Every possible combination of inputs? Which interactions cause faults? Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  5. Combinatorial approach • Pairwise combination instead of exhaustive • Generate combinations that efficiently cover all pairs of values • Extended by t-wise: test all the combinations of t values • Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs (triples,…) reduces the number of test cases, but reveals most faults Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  6. Example • 3 variables with 3 values each: 33 = 27 possible combinations • Combinatorial testing with much fewer tests Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  7. Test Suite • pairwise testing can be achieved by only 9 tests One test covers many combinations: e.g. Test 1 covers 3 pairs: (Monochrome, Full-graphics) (Monochrome, Hand-held) (Full-graphics, Hand-held) • 2100 combinations with 10 tests; 1020200 tests; … Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  8. CIT effectiveness • Experiments show that CIT is • effective • finds faults that traditional testing may be not able to find • efficient • A low degree of interaction between inputs can already discover most faults • Pairwise is the most used • Never with interaction > 6 Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  9. Methods for building CIT test suites • Methods and tools • IPO, AETG, PICT, TestCover, … • www.pairwise.org • Main goal: producing small test suites • Because the problem of generating a minimum test suit for combinatorial testing is NP-complete, most methods and tools use a greedy approach Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  10. Adding constraints • Constraints express relations among inputs: a valid test must not violate the constraints • E.g. if the display mode is in text-only, the color is monochrome •  Constrained CIT • Only few test generation methods support constraints, all of them only as “forbidden tuples” • Goal: supporting expressive constraints • Easier to obtain from the requirements Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  11. Logic approach to CCIT [tap08] • formalize pairwise testing: express each pair as a corresponding logical expression, a test predicate • p1 = v1 /\ p2 = v2 • p1 and p2 are inputs, v1 and v2 their values • Similarly, the t-wise coverage can be modeled by a set of test predicates, each of the type • p1 = v1 /\ p2 = v2 /\ … /\ pt = vt Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  12. Test = logical model • test: an assignment to all the inputs of the system • a test ts covers a test predicate tp if and only if it is a model of tp: • ts |= tp • Finding a test suite  finding a model for each test predicate Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  13. Adding constraints • With the constraints c1,…, cn, a test is a model of tpand the constraintsc_i • ts |= tp /\ c1 /\ … /\ cn • the constraints become first class citizens and they can be represented by logical expressions too • Finding tests becomes a problem of finding a model of a complex expression • many techniques like, constraint solvers, model checkers [AFM08], SAT solver, SMT solvers … Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  14. Yices[csl.sri.com] • Efficient SMT solver that decides the satisfiability of arbitrary formulas • Expressive language for constraints and input models • Powerful algorithms to find models Test predicate Model = test YICES constraints Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  15. Test Suite generation a simplified version Test predicates Test suite builder Test predicate Model= test Input model YICES constraints Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  16. Monitoring and reduction • Monitoring: A test covers may cover several test predicates which can be removed from the pool • Reduction: at the end, a minimal set of tests that cover all the test predicates can be found by a greedy algorithm t1 t2 Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  17. Collecting Model = test that covers all the test predicates collected Instead of one test for every tp, collect the tps to build a conjoint Collected test predicates tp1 /\ tp2 /\ tp3 Test suite builder YICES Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  18. Ordering • In which order test predicates can be collected? • First simple two policies: • the order in whichtp are generated • random order Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  19. Random Ordering • non deterministic ordering • Like most CIT approaches: run the method 50 times and then take the best result Best result = min test suite Worst result = max test suite • GOAL: finding a deterministic ordering that performs as better as the random ordering Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  20. Deterministic Ordering Policies • Several criteria for ordering the test predicates can be defined • Looking at the tests already generated and at the test predicates still to cover Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  21. Deterministic Ordering Policies /c variants Consider not only the tests in the test suite (test already generated) but also the other test predicates in the collected test predicate Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  22. Case Studies • 13 specifications with constraints taken from the literature • From 3 ^3 to 8.3 x 10^6 combinations • From 6 to 50 constraints (forbidden tuples) Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  23. Experimental Results • Among policies: • One policy (touch/c) performed always better except in two cases in which touch performed better • As generated: bad policy Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  24. Comparison with random random max Bad policies: worse than average of random ordering avg Good policies: better than average of random ordering min Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  25. Comparison with random Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

  26. Conclusions • Logic approach can be applied to constrained combinatorial testing • It allows a more natural (and expressive) formalization of constraints • Good heuristics are defined to order the test predicates and obtain small test suites • Random ordering still useful in case of unlimited resources • Prototype tool available at http://cs.unibg.it/gargantini/software/atgt/ Calvagna & Gargantini - Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

More Related