Testing your Software

1. Testing your Software Joe Meehean

2. Testing is the process of executing a program with the intent of finding errors. -Glenford Myers

3. Testing • Testing is done to find errors • not to prove there are none • successful tests find errors • Any useful program has errors • you won’t be able to find them all • find and fix the big ones

4. Black Box Testing • Program is an opaque box • ignore internal algorithms or structure • act like someone else wrote it • give it to someone else to test • Provide a variety of inputs,check for correct output • seems simple, its not

5. Black Box Testing • Problem: • Even simplest programs can have billions of possible inputs • cannot possibly test them all • must choose a useful subset • How do we do this methodically?

6. Black Box Testing … -2 -1 0 1 …… 12 13 14 15 … Invalid Partition 1 Invalid Partition 2 Valid Partition • Equivalence partitioning • partition input into sets • provide a test for each set • Month # to Month Name Program

7. Black Box Testing … -2 -1 0 1 …… 12 13 14 15 … Invalid Partition 1 Invalid Partition 2 Valid Partition • Boundary-value analysis • select inputs from boundary of equivalence partitions • where inputs switch from one set to another

8. Black Box Testing • All pairs testing • for exactly 2 input arguments only • test all combinations of those inputs • or all combination of boundary values • Intuition • most common errors involve 1 argument • next most common involve 2 • … • 2 highest we can go without prohibitive cost

9. Black Box Testing • Fuzz testing • create a program that generates random inputs • feed random input into program under test • Monitor tested program for • crashes • unhandled exceptions • wedging (hanging) • Term coined by Barton Miller

10. Black Box Testing Model Model Input Model Output Test Idea Compare Real Input Program Real Output • Model-based testing • create a mathematical model of your program • create real test cases and abstract test cases • compare the results of real run with abstract run

11. Black Box Testing • Model-based testing • runtime complexity can be a model • not the only model • E.g., you expect your program to be O(N) • run your program with input size X • use your model to calculate runtime for size 2X • run program with size 2X • compare results

12. Black Box Testing • Exploratory Testing • play with it to see how it works • knowing how it works, try to break it • “it looks like it works like this” • “what if I try this” • make fun of a program until it cries

13. Black Box Testing • How/when to use these techniques? • Equivalence partitioning • ALWAYS • Boundary-value analysis • ALWAYS • All-pairs • as needed • if your program takes pairs of inputs

14. Black Box Testing • How/when to use these techniques? • Fuzz-testing • in school, try this by hand (make up random) • at work, write software to support • Model-based • when it makes sense • is your program behaving strangely,but still producing correct output?

15. Black Box Testing • How/when to use these techniques? • Exploratory • ALWAYS

16. Questions?

17. White Box Testing • Create tests to evaluate source code • specific functions • specific lines • specific conditions • Most common kind of testing done by developers

18. White Box Testing • API Testing • Application Programming Interface (API) • ensure every public & private method does what it should • using either black box (method is black box)or other white box techniques • can add testing code directly to program

19. White Box Testing • API Testing • e.g., test LCVector::doubleCapacity() • does it double the capacity? • does it copy all of the items over to the new array? • print raw array before double capacity • print raw array after double capacity

20. White Box Testing • Code coverage • technique to evaluate black box tests • ensure every piece of code executed once • requires software support • Types • function coverage • statement coverage • condition coverage • every boolean sub-expression of a condition • test for both true and false

21. White Box Testing • Fault injection • What is a fault? • unexpected event or condition • faults may cause errors • e.g., disk drive fails, memory corrupted, … • What is an error? • output that doesn’t meet spec • crashing • hung

22. White Box Testing • Fault injection • artificially cause faults at runtime • see what the program does • try to inject faults at specific times • e.g., modify OS to pretend disk crash during file read • observe the programs behavior • kind of like extra mean fuzz testing

23. White Box Testing • Testing all parts of your program doesn’t guarantee it is bug-free • Program may be missing code • Won’t find data-sensitivity errors • covers all lines of code • not all possible inputs

24. White Box Testing • How/when to use these techniques? • API testing • ALWAYS • test it as you build it • Code coverage • when tools are available • and not time prohibitive • this may be a requirement of your job

25. White Box Testing • How/when to use these techniques? • Fault injection • when your program must work all the time • graduate school • designing robust, mission-critical systems • e.g., autopilot software

26. Questions?