Orthogonal Arrays Testing Strategy (OATS)

1. Orthogonal ArraysTestingStrategy(OATS) Mohamed Fathy QA Engineer Arx ICT September 4, 2009

2. Agenda • OATS Overview. • How to use Orthogonal Arrays? • OATS Examples. • Case Study. • Where OATS could be applied? • OATS Benefits. • References & Resources. • Questions?

3. OATS Overview • Taguchi's orthogonal arrays are highly fractional orthogonal designs proposed by Dr. Genichi Taguchi. • OATS is one such combinatorial testing method. • OATS is a systematic, statistical way of testing pair-wise interactions.

4. OATS Overview (Cont’d) • It addresses generation of test cases for problems involving multiple parameters and combinations. • It can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases.

5. How to use Orthogonal Arrays? Lruns(Levels^Factors)  or Ln(s^k): Orthogonal array of N runs where k factors take on any value from a set of s symbols/level. • Runs: The number of rows or number of test cases in the array that will be generated by Orthogonal Array technique. Each row represents a test case. • Factors: The number of columns or the number of parameters/variables in an array that need to be tested in the System. • Levels: The maximum number of values in an Orthogonal Array that can be taken on by any single factor.

6. OATS Examples Example 1

7. OATS Examples (Cont’d) The borders and shading dialog box : • 5 settings • 5 styles (showing) • 9 colors • 9 widths 5 x 5 x 9 x 9 = 2025 combinations! L81(94) array will suffice

8. OATS Examples (Cont’d) Example 2

9. OATS Examples (Cont’d) • 3 toolbars (pictures, text, or both). • 3 choices for launch on startup (browser, mail or news). • 3 choices for startup page (blank home page, home page names a valid file, or home page name has a syntax error). • 2 choices for underlining of links (don’t underline, or underline). • 2 choices for expiration of links (never expire, expire after 30 days). 3 x 3 x 3 x 2 x 2 = 108 combinations!

10. OATS Examples (Cont’d)

11. OATS Examples (Cont’d)

12. OATS Examples (Cont’d)

13. OATS Examples (Cont’d) • Example 3:

14. OATS Examples (Cont’d) • Orthogonal array to be used: L9 0000 0112 0221 1011 1120 1202 2022 2101 2210

15. Case Study

16. Optimization • Remove invalid runs. • Optimization need not necessarily mean the reduction of tests. You can add some combinations that are critical for testing, but missed by OATS.

17. Where OATS could be applied? OATS could be applied in: • User interface testing. • System testing. • Regression testing. • Configuration testing. • Performance testing. • This method is extremely valuable for testing complex applications. • Useful for testing products with a large number of configuration possibilities.

18. Benefits of OATS • Provides uniformly distributed coverage of the test domain. • Concise test set with fewer test cases is created. • All pair-wise comibnations of test set created. • Arrives at complex combinations of all the variables.

19. Benefits of OATS (Cont’d) • Simpler to generate and less error prone than test sets created manually. • Reduces testing cycle time.

20. References & Resources • References: 1- R Krishnan, S Murali Krishna, P Siva Nandhan - Motorola: Combinatorial Testing: Learnings from our Experience . 2-Madhav S. Phadke , Quality Engineering Using Robust Design. • Resources: 1- http://www.research.att.com/~njas/oadir/index.html 2-http://www.mathdox.com/nguyen/orthogonal_array.jsp

21. Questions? Thank You Feedback: mfathy@arxict.com