Simplified Modeling of Combinatorial Test Spaces

1. Simplified Modeling of Combinatorial Test Spaces Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs

2. Agenda • Introduction • Counters • Properties • Implementation • Summary

3. Background – Restrictions • Restrictions are a crucial part of real-life combinatorial models • Effort and expertise required for capturing the restrictions is high • Existing work concentrates on the ability to state complex restrictions • Typically as propositional formulas or predicates • Can be captured using classification trees

4. The Problem Restrictions are typically: • Cumbersome to read and review • Not very generalized (exclude specific conditions) • Hard to maintain

5. Introduction • An approach for handling some complex restrictions • We introduce two new constructs to the CTD model: • Counter parameters • Value properties • Real-life case studies

6. Counters

7. Counters – Example • Consider testing an upgrade process for a virtualized server • Traverses the host and VMs • Upgrades various out-of-date components • Ten VMs, each has one of four OS versions: RedHat 5.7, RedHat 6.1, AIX 6.1.3, AIX 6.1.6 • 2 versions are marked “preferred” • Triggered when at least half are out of date

8. Counters – Example – Cont. • Parameters and values: • VM1 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6 … • VM10 – RedHat5.7 / RedHat6.1 / AIX6.1.3 / AIX6.1.6

9. Counters – Example – Cont. “Triggered when at least half are out of date”… ?!

10. Counters – Example – Cont. • (Bad) option I: explicitly exclude all combinations in which less than half are out of date • All 653,312 of them !

11. Counters – Example – Cont. • (Bad) option II: choose 5 VMs, and force them to be outdated • VM1 != “AIX6.1.6” && VM1 != “RedHat6.1” && VM2 != “AIX6.1.6” && VM2 != “RedHat6.1” && …VM5 != “AIX6.1.6” && VM5 != “RedHat6.1” • 252 such (terrible) restrictions

12. Counters • A counter is a parameter that counts values in other parameters • Automatically evaluated in each test • Can be used in restrictions and in coverage requirements

13. Counters – Back to the example • Define counter “preferred” • Counts appearances of “RedHat6.1” and “AIX6.1.6” in parameters VM1 … VM10 • One restriction that excludes: preferred > 4

14. Auxiliary Variables • The notion can be extended to auxiliary variables • Parameters that do not define the test • but are rather a function of the defining parameters • Examples: • Sum of other numeric parameters • Count the number of duplicate values

15. (Touching on) Implementation • Naïve implementation: Add the counter to the model, and restrictions that correctly map its values to the counted values • Much better: Represent as functions on other parameters • More details in the paper

16. Properties

17. Properties – Example • Consider testing a Storage Area Network (SAN) • A server (one of four types – two x86-based, two PowerPC-based) • An operating system (one of 17 versions – two Windows-based, three AIX-based, one virtualized, others Linux-based) • HBA (one of 11 types, of different manufacturers)

18. Properties – Example – Cont. • Some OSs do not run on some processors (e.g., AIX on x86, Windows on PowerPC) • Some HBAs do not have drivers for some OSs • Some HBAs cannot connect to some processors • A total of 299 pairs to be excluded • Tedious and error-prone

19. Properties • Properties may be defined for parameters • A value to the property is assigned for each value of the parameter • Can be used in restrictions and in coverage requirements • Represent different abstractions of the parameters

20. Properties – Back to the example • Define property “Architecture” for parameter “Server” • The two PowerPC servers will have value PowerPC • The two x86 servers will have value x86 • Define property “OSFamily” for parameter “OS”, with values Windows, AIX, Linux

21. Properties – Back to the example – cont. • Define properties “X_Compatible”, “P_Compatible”, “AIX_Compatible” and “Win_Compatible” for the HBA parameter • Give property values to the parameter values according to their compatibility with x86, PowerPC, AIX and Windows, respectively

22. Properties – Back to the example – Cont. • The restrictions now disallow: • Server.Architecture == “PowerPC” && OS.OSFamily == “Windows” • Server.Architecture == “x86” && OS.OSFamily == “AIX” • Server.Architecture == “PowerPC” && ! HBA.P_Compatible • Server.Architecture == “x86” && ! HBA.X_Compatible • OS.OSFamily == “AIX” && ! HBA.AIX_Compatible • OS.OSFamily == “Windows” && ! HBA.Win_Compatible

23. Simplified Modeling – Benefits • More readable restrictions • Easier to review • More general • Preserves time, reduces risk of errors • More maintainable • e.g., a new Windows-based OS is introduced

24. Implementation • Our symbolic BDD-based implementation is agnostic to test-space representation • It merely requires one to symbolically represent the set of valid tests • Simple to extend to support counters & properties • More details in the paper

25. Summary • Correctly capturing restrictions is a significant obstacle in combinatorial testing deployment • We introduced two constructs that significantly simplify restrictions in some cases • Examples based on real-life cases

26. Thank You For Listening