Combinatorial Methods in Software Testing Tutorial

1. Combinatorial Methods in Software TestingRick KuhnNational Institute of Standards and Technology Gaithersburg, MD East Carolina University NSF Research Experiences for Undergraduates June 29, 2015

2. Tutorial Overview • Why are we doing this? • What is combinatorial testing? • What tools are available? • Is this stuff really useful in the real world? • What's next?

3. What is NIST and why are we doing this? • US Government agency, whose mission is to support US industry through developing better measurement and test methods • 3,000 scientists, engineers, and staff including 4 Nobel laureates • Project goal – improve cost-benefit ratio for testing

4. What good is it? • Cooperative R&D Agreement w/ Lockheed Martin • 2.5 year study, 8 Lockheed Martin pilot projects in aerospace software • Results announced Feb. 2013; IEEE Computer paper April, 2015 • Results: “Our initial estimate is that this method supported by the technology can save up to 20% of test planning/design costs if done early on a program while increasing test coverage by 20% to 50%.” • Note that testing is typically 50% of software cost

5. What good is it (cont.)? • Rockwell Collins “used this approach to automate parts of the unit and integration testing of a 196 KSLOC avionics system.” • “The goal was to see if it might cost-effectively reduce rework by reducing the number of software defects escaping into system test” • “Overcoming scalability issues required moderate effort, but in general it was effective – e.g., generating 47,040 test cases (input vectors, expected outputs) in 75 seconds, executing and analyzing them in 2.6 hours. It subsequently detected all seeded defects, and achieved nearly 100% structural coverage.” • According to NASA, ratio of test cost to development cost for life-critical avionics roughly 7:1

6. Software Failure Analysis • NIST studied software failures in a variety of fields including 15 years of FDA medical device recall data • What causes software failures? • logic errors? • calculation errors? • inadequate input checking? • interaction faults? Etc. Interaction faults: e.g., failure occurs ifpressure < 10 && volume>300 (interaction between 2 factors) Example from FDA failure analysis: Failure when “altitude adjustment set on 0 meters and total flow volume set at delivery rate of less than 2.2 liters per minute.” So this is a 2-way interaction

7. ● Computer Security Division ● Additional examples (from National Vulnerability Database) • Single variable, 1-way interactionexample: Heap-based buffer overflow in the SFTP protocol handler for Panic Transmit … allows remote attackers to execute arbitrary code via a long ftps:// URL. • 2-way interactionexample: single character search string in conjunction with a single character replacement string, which causes an "off by one overflow" • 3-way interactionexample: Directory traversal vulnerability when register_globals is enabled and magic_quotes is disabled and .. (dot dot) in the page parameter

8. Interaction Fault Internals How does an interaction fault manifest itself in code? Example: altitude_adj == 0 && volume < 2.2 (2-way interaction)‏ if (altitude_adj == 0 ) { // do something if (volume < 2.2) { faulty code! BOOM! } else { good code, no problem} } else { // do something else } A test with altitude_adj == 0 and volume = 1 would find this ~ 90% of the FDA failures were 2-way or 1-way

9. How are interaction faults distributed? • Interactions e.g., failure occurs if pressure < 10 (1-way interaction)‏ pressure < 10 & volume > 300 (2-way interaction)‏ pressure < 10 & volume > 300 & velocity = 5 (3-way interaction)‏ • Surprisingly, no one had looked at interactions beyond 2-way before • The most complex medical device failure reported required a 4-way interaction to trigger. Interesting, but that's just one kind of application! Number of factors involved in faults

10. What about other applications? Server (green) These faults more complex than medical device software!! Why? Number of factors involved in faults

11. Others? Browser (magenta) Number of factors involved in faults

12. Still more? NASA Goddard distributed database (light blue) Number of factors involved in faults

13. Even more? FAA Traffic Collision Avoidance System module (seeded errors) (purple) Number of factors involved in faults

14. Finally Network security (Bell, 2006) (orange) Curves appear to be similar across a variety of application domains. Number of factors involved in faults

15. What causes this distribution? One clue: branches in avionics software.7,685 expressions from if and while statements

16. Comparing with Failure Data Branch statements Distribution of t-way faults in untested software seems to be similar to distribution of t-way branches in code

17. Number of factors involved in faults • Number of factors involved in failures is small • New algorithms make it practical to test these combinations • We test large number of combinations with very few tests

18. Interaction Rule • So, how many parameters are involved in faults? Interaction rule: most failures are triggered by one or two parameters, and progressively fewer by three, four, or more parameters, and the maximum interaction degree is small. • Maximum interactions for fault triggering was 6 • Reasonable evidence that maximum interaction strength for fault triggering is relatively small So, how does it help me to know this?

19. How does this knowledge help? • If all faults are triggered by the interaction of t or fewer variables, then testing all t-way combinations can provide strong assurance • We can do this using combinatorial methods • But still need to consider value propagation issues, equivalence partitioning, timing issues, more complex interactions, . . . Still no silver bullet. Rats!

20. Complications • Code block faults: • if (correct condition) {faulty code} • else {correct code} • Condition faults: • if (faulty condition) {correct code} • else {correct code} • Condition faults are harder to detect than code block faults(but consider nested conditionals, the usual case)

21. Detection of condition faults Balance, Vilkomir, 2012 How do nested conditionals combine to produce overall t-way fault distribution seen previously?

22. Tutorial Overview • Why are we doing this? • What is combinatorial testing? • What tools are available? • Is this stuff really useful in the real world? • What's next?

23. Where did these ideas come from? • Scottish physician James Lind determined cure for scurvy • Ship HM Bark Salisbury in 1747 • 12 sailors “were as similar as I could have them” • 6 treatments 2 sailors for each – cider, sulfuric acid, vinegar, seawater, orange/lemon juice, barley water • Principles used (blocking, replication, randomization) • Did not consider interactions, but otherwise used basic Design of Experiments principles

24. Early Design of Experiments methods Used in 1920s for agriculture research • Key features of DoE • Blocking • Replication • Randomization • Orthogonal arrays to test interactions between factors Each combination occurs same number of times, usually once. Example: P1, P2 = 1,2 Sounds great!Let’s use it for software!

25. Orthogonal Arrays for Software Testing • Functional (black-box) testing • Hardware-software systems • Identify single and 2-way combination faults • Early papers • Taguchi followers (mid1980’s) • Mandl (1985) Compiler testing • Tatsumi et al (1987) Fujitsu • Sacks et al (1989) Computer experiments • Brownlie et al (1992) AT&T • Generation of test suites using OAs • OATS (Phadke, AT&T-BL) Results good, but not great.

26. What’s different about software? Traditional DoE • Continuous variable results • Small number of parameters • Interactions typically increase or decrease output variable • 2-way interactions studied DoE for Software • Binary result (pass or fail) • Large number of parameters • Interactions affect path through program • 2-way to 6-way interactions matter

27. NIST What do these differences mean for testing software? • Don’t use orthogonal arrays, use covering arrays • Cover every t-way combination at least once • Key differencesorthogonal arrays:covering arrays: • Combinations occur at least once • Always possible to find for a particular configuration • Size always ≤ orthogonal array • Combinations occur same number of times • Not always possible to find for a particular configuration

28. Let’s see how to use this knowledge in testing. A simple example:

29. How Many Tests Would It Take? • There are 10 effects, each can be on or off • All combinations is 210 = 1,024 tests • What if our budget is too limited for these tests? • Instead, let’s look at all 3-way interactions …

30. 10 3 Now How Many Would It Take? • There are = 120 3-way interactions. • Naively 120 x 23 = 960 tests. • Since we can pack 3 triples into each test, we need no more than 320 tests. • Each test exercises many triples: 0 1 1 0 0 0 0 1 1 0 OK, OK, what’s the smallest number of tests we need?

31. 10 3 A covering array All triples in only 13 tests, covering 23 = 960 combinations Each column is a parameter: Each row is a test: • Developed 1990s • Extends Design of Experiments concept • NP hard problem but good algorithms now

32. A larger example Suppose we have a system with on-off switches. Software must produce the right response for any combination of switch settings:

33. How do we test this? 34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests

34. What if we knew no failure involves more than 3 switch settings interacting? • 34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests • If only 3-way interactions, need only 33 tests • For 4-way interactions, need only 85 tests

35. 33 tests for this range of fault detection (on average) 85 tests for this range of fault detection (on average) That’s way less than 17 billion! Number of factors involved in faults

36. Tutorial Overview • Why are we doing this? • What is combinatorial testing? • What tools are available? • Is this stuff really useful in the real world? • What's next?

37. Available Tools • Covering array generator – basic tool for test input or configurations; • Sequence covering array generator – new concept; applies combinatorial methods to event sequence testing • Combinatorial coverage measurement – detailed analysis of combination coverage; automated generation of supplemental tests; helpful for integrating c/t with existing test methods • Domain/application specific tools: • Access control policy tester • .NET config file generator

38. IPOG ITCH (IBM)‏ Jenny (Open Source)‏ TConfig (U. of Ottawa)‏ TVG (Open Source)‏ T-Way Size Time Size Time Size Time Size Time Size Time 2 100 0.8 120 0.73 108 0.001 108 >1 hour 101 2.75 3 400 0.36 2388 1020 413 0.71 472 >12 hour 9158 3.07 4 1363 3.05 1484 5400 1536 3.54 1476 >21 hour 64696 127 4226 NA 18s >1 day 5 4580 43.54 NA >1 day 313056 1549 6 10941 65.03 NA >1 day 11625 470 NA >1 day 1070048 12600 How do we generate these arrays? • Greedy algorithm • Supports constraints among variable values • Freely available Traffic Collision Avoidance System (TCAS): 273241102 Times in seconds

39. ACTS - Defining a new system

40. Variable interaction strength

41. Constraints

42. Covering array output

43. Output options Mappable values Degree of interaction coverage: 2 Number of parameters: 12 Number of tests: 100 ----------------------------- 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 1 1 1 1 2 0 1 0 1 0 2 0 2 2 1 0 0 1 0 1 0 1 3 0 3 1 0 1 1 1 0 0 0 1 0 0 4 2 1 0 2 1 0 1 1 0 1 0 5 0 0 1 0 1 1 1 0 1 2 0 6 0 0 0 1 0 1 0 1 0 3 0 7 0 1 1 2 0 1 1 0 1 0 0 8 1 0 0 0 0 0 0 1 0 1 0 9 2 1 1 1 1 0 0 1 0 2 1 0 1 0 1 Etc. Human readable Degree of interaction coverage: 2 Number of parameters: 12 Maximum number of values per parameter: 10 Number of configurations: 100 ----------------------------------- Configuration #1: 1 = Cur_Vertical_Sep=299 2 = High_Confidence=true 3 = Two_of_Three_Reports=true 4 = Own_Tracked_Alt=1 5 = Other_Tracked_Alt=1 6 = Own_Tracked_Alt_Rate=600 7 = Alt_Layer_Value=0 8 = Up_Separation=0 9 = Down_Separation=0 10 = Other_RAC=NO_INTENT 11 = Other_Capability=TCAS_CA 12 = Climb_Inhibit=true

44. Who uses combinatorial testing? Telecom Defense Finance Information Technology

45. How much does it cost? • Number of tests: proportional to vt log n • for v values, n variables, t-way interactions • Tests increase exponentially with interaction strength t • But logarithmically with the number of parameters • Example: suppose we want all 4-way combinations of n parameters, 5 values each:

46. Is this stuff actually useful in the real world ??

47. Foundation – interaction rule • Rule says that all failures are triggered by a small number of factors interacting; empirical data says 1 to 6 • Therefore, if we cover all t-way combinations, for small t, we should have testing that is “pseudo- exhaustive” • Does this really work?

48. Example 1: Document Object Model Events • DOM is a World Wide Web Consortium standard for representing and interacting with browser objects • NIST developed conformance tests for DOM • Tests covered all possible combinations of discretized values, >36,000 tests • Question: can we use the Interaction Rule to increase test effectiveness the way we claim?

49. Document Object Model Events Original test set: Exhaustive testing of equivalence class values

50. Document Object Model Events Combinatorial test set: All failures found using < 5% of original exhaustive test set