Testing software

1. Testing software Team Software Development Project Module Web Page: http://www.comp.dit.ie/bmacnamee/tsdp.htm

2. Why testing ? • Would you trust a completely automated nuclear power plant ? • Would you trust a completely automated pilot whose software was written by yourself ? What if it was written by your class ? • Would you dare to write an expert system to diagnose cancer. What if you are personally held liable in cases where a patient dies because of a malfunction of the software? • All software has faults!

3. Poor testing can cost lives • The Mars Climate Orbiter crashed in September 1999 because of a "silly mistake": wrong units in a program. • The 1988 shooting down of the Airbus 320 by the USS Vincennes was attributed to the cryptic and misleading output displayed by the tracking software. • Several 1985-7 deaths of cancer patients were due to overdoses of radiation resulting from a race condition between concurrent tasks in the Therac-25 software. • Errors in medical software have caused deaths. Details in B.W. Boehm, "Software and its Impact: A Quantitative Assessment," Datamation, 19(5), 48-59(1973). • A China Airlines Airbus Industrie A300 crashes on April 26, 1994 killing 264. Recommendations include software modifications.

4. Why testing ? • "perfect" software doesn't exist • All released software is imperfect • Software is complex & software engineering is an immature field • 60% of errors introduced during design, 40% during implementation • Cost of error-correction goes up with life cycle stage • 2/3 of design errors are discovered when the software is operational. • Testing identifies failures and faults in the software

5. Some testing principles • A programmer should not test his/her own program. • One should test not only that the program does what it is supposed to do, but that it does not do what it is not supposed to. • No amount of testing can guarantee error-free program. • Parts of programs where a lot of errors have already been found are a good place to look for more errors. • The goal is not to embarrass the programmer!

6. Before testing starts:Inspections and Walkthroughs • Inspections and walkthroughs should be carried out throughout the development process. • Done by a team of people. • Performed at a meeting • The programmer explains the program line by line. • Questions are raised. • Can find 30%-70% of errors. • NB: Extreme programming uses pair programming to build this into the core development process

7. Data declaration All variables declared? Default values understood? Arrays and strings initialized? Variables with similar names? Correct initialization? Control flow Each loop terminates? Are errors handled? Input/output Format specification correct? Are invalid inputs handled? Checklist for inspections

8. Types of tests requirements Acceptance Test More likely to be GUI based Analysis System Test Design Integration Test Mostly requires small test programmes to be written Implementation Unit Test Each phase of the development life-cycle must be tested

9. Test strategies • Testing only shows the presence or errors, not the absence. Exhaustive testing is impossible with most software • test all valid inputs • test all invalid inputs • test all edited inputs • test all variations on input timing • Proving program correctness is almost impossible with most software • There need to balance the level of certainty of correctness against cost and impact of faults.

10. Classification of test techniques • According to the criterion to measure the adequacy of the set of test cases: • Coverage based testing • Fault-based testing • Error-based testing • Based on the source of information used to derive test cases • Black Box Testing (functional, specification-based) • White Box Testing (structural, program-based) • All tests require a clear description of what the test does and what result is expected. • i.e. Enter numbers 12345 in name field and press okay. Expect a ‘please type in your name again’ • i.e. pass value -1 for length parameter into GetNewAccount function. Expect error ABC to be thrown.

11. Test Objectives • Tests identify failures or faults in the software. • Failures are ways in which the software fails to run according to requirements. • Note that some requirements will be implicit: Safety requirements may not be stated! • Failures are caused by faults which are caused by errors in the developed software. • 1 error may lead to many faults • A failure may occur a "long way" from the original fault • One fault may lead to many failures

12. Black-Box testing • The testing is not based on the structure of the program (which is hidden/unknown). • Theoretically, in order to ensure correctness, every possible input needs to be tested • this is impossible. • The goal: to maximize the number of errors found for the available effort available for testing.

13. White-Box testing • Is based on the internal structure of the program. • There are several alternative criteria for checking “enough” paths in the program. • Even checking all paths within a component does not guarantee finding all errors. • Does not cover prior and post implications of errors.

14. Selection of test cases (for white-box testing) • Select a good coverage criterion. Completeness would require: • Cover all paths of the program. • Execute every statement at least once. • Each decision has a true or false value at least once. • Each condition is taking each truth value at least once. • Check all possible combinations of conditions in each decision. • For this project, • Select key components of the software for white-box testing • Select key paths for detailed testing. • Focus on black-box testing for the rest.

15. Identifying black-box test cases • Partition the program into pieces of code with a single entry/exit point. • For each piece find which variables are set/used/tested. • Create test cases to check that the piece works according to expectation. • E.g. handles unexpected values passed in, functions called out of order etc.

16. Analysing test results • Once testing has discovered failures and faults, they need to be classified by impact on the software. e.g. • Type A: System failure – a significant component of the software does not work causing the whole system to fail. • Type B: Module failure – a sub-set of the software does not work, but does not impact the overall operation of the software • Type C: Minor failure – some non-core functionality does not work according to specification. • Type D: Cosmetic issue: e.g. User Interface issues

17. Release criteria • Before testing commences release criteria are set by the QA manager which need to be met before the software can be released: • Software should never be released with type A failures/faults. • Where faults/failures exist in the released software, either • They need to documented in release notes • The functionality associated needs to be removed from the release

18. Test plan/results document • For this module, the test plan and test results documents have been combined. • All projects must complete the plan and results sections as part of their completed project. • Note: This is very simplified version of what is actually be used.