Software Testing and Quality Assurance

1. Software Testing and Quality Assurance Motivation and Review of Software Verification & Validation

2. Reading • Ron Patton, Software Testing, 2nd ed., 2006. • Chapter 1 • Daniel Michaels and Andy Pasztor, “Incidents Prompt New Scrutiny Of Airplane Software Glitches”, Wall Street Journal, Vol. 247, no. 125, May 30, 2006. • Safety Critical Overview • Somerville, Ian. Software Engineering, 7th Edition, Addison-Wesley, 2004. • Chapter 22

3. Outline • Motivation • Introduction to software verification and validation concepts. • Software verification and validation process. • Introduction to static & dynamic verification analysis techniques. • Software Inspections • Software Testing

4. Motivation (1) • Why Software Testing? • The Lion King Animated Storybook, Fall 1994 • Disney’s first multimedia CD-ROM game for children. • Sales were huge. • The game to buy for children that holiday season. • On December 26, Disney's customer support phones began to ring, and ring, and ring….from angry parents with crying children who couldn't get the software to work. Numerous stories appeared in newspapers and on TV news. • The software development team have tried their software on specific PC platforms. It failed on many very popular PC platforms!

5. Motivation (2) • Why Software Testing? • Intel Pentium Floating-Point Division Bug, 1994 • Enter the following equation into your PC's calculator: (4195835 / 3145727) * 3145727 - 4195835 If the answer is zero, your computer is just fine. If you get anything else, you have an old Intel Pentium CPU with a floating-point division bug, a software bug burned into a computer chip and reproduced over and over in the manufacturing process. • On October 30, 1994, Dr. Thomas R. Nicely of Lynchburg (Virginia) College traced an unexpected result from one of his experiments to an incorrect answer by a division problem solved on his Pentium PC. He posted his finding on the Internet and soon afterward a firestorm erupted as numerous other people duplicated his problem and found additional situations that resulted in wrong answers. • Fortunately, these cases were rare and resulted in wrong answers only for extremely math-intensive, scientific, and engineering calculations.

6. Motivation (3) • Why Software Testing? • NASA Mars Polar Lander, 1999 • On December 3, 1999, NASA's Mars Polar Lander disappeared during its landing attempt on the Martian surface. • A Failure Review Board investigated the failure and determined that the most likely reason for the malfunction was the unexpected setting of a single data bit. • Most alarming was why the problem wasn't caught by internal tests.

7. Motivation (3) • Why Software Testing? • Malaysia Airlines jetliner, August 2005 • As a Malaysia Airlines jetliner cruised from Perth, Australia, to Kuala Lumpur, Malaysia, it suddenly took on a mind of its own and zoomed 3,000 feet upward. • The captain disconnected the autopilot and pointed the Boeing 777’s nose down to avoid stalling, but was jerked into a steep dive. He throttled back sharply on both engines, trying to slow the plane. Instead, the jet raced into another climb. • The crew eventually regained control and manually flew their 177 passengers safely back to Australia. • Investigators quickly discovered the reason for the plane’s roller-coaster ride 38,000 feet above the Indian Ocean. A defective software program had provided incorrect data about the aircraft’s speed and acceleration, confusing flight computers. The computers had also failed, at first, to respond to the pilot’s commands.

8. Motivation (4) • Why Software Testing? • http://www.aonix.com/safety_critical_overview.html • A passenger airplane is circling in a prearranged location off the coast of Florida. The landing is delayed because of bad weather conditions. As the plane is banking into a turn, a sudden updraft causes the plane to roll much faster than the software control system expects. The software "assumes" a glitch, and the computers are set into an automatic reboot process. The pilot looks on with horror as all of the navigation displays turn blue with a white line through them. At a most crucial moment, when the pilot needs information to stabilize the aircraft, the computers are performing memory checks and restarting the display software.

9. Verification & Validation (V &V) Process • V & V takes place at each phase of software development life cycle. • Requirements reviews. • Design reviews. • Code inspections to software testing. • Has two principal objectives • The discovery of defects in a system. • The assessment of whether or not the system is useful and useable in an operational situation.

10. Validation • Are we building the right product? • The process of evaluating a system during and at the end of the development process to determine if it satisfies the requirements of the system. • The software should do what the user really requires.

11. Verification • Are we building the product right? • The process of evaluating a system at the end of a phase to determine if it satisfies the conditions imposed at the start of that phase. • The software should conform to its specification.

12. V & V goals • Ultimate goal • Software is ‘fit for the purpose’. • Software is not necessarily 100 % free of defects. • Rather, it must be good enough • for its intended use; and • the type of use will determine the degree of confidence that is needed.

13. V & V confidence • Depends on system’s purpose, user expectations and marketing environment • Software function • The level of confidence depends on how critical the software is to an organization. • User expectations • Users may have low expectations of certain kinds of software. • Marketing environment • Getting a product to market early may be more important than finding defects in the program.

14. Two Types of Verification (1) • Static Verification • Software inspections • Concerned with analysis of the static system representation to discover problems. • Review the documents and software system during different phases of development life-cycle. • Supplement by tool-based document and code analysis.

15. Two Types of Verification (2) • Dynamic Verification • Concerned with excising and observing product behavior • The system is executed with test data and its operational behavior is observed.

16. Static and dynamic V & V Software Inspections Requirements Specification High-level Design Formal Specification Detailed Design Program Program Testing Prototype

17. Software Inspections • Involves people examining the software documentation • With the aim of discovering anomalies and defects. • Inspections do not require execution of a system • Can be used before implementation.

18. Software Inspections Without Inspection No. of Employees With Inspection Requirements Planning Design Coding Testing Time

19. Inspection Success • Different defects may be discovered in a single inspection. • In testing, one defect may mask another so several executions are required. • Reuse of domain and programming knowledge • Common mistakes and errors types.

20. Program Testing • Can reveal the presence of errors • Not their absence • The only V&V technique for non-functional requirements • As the software has to be executed to see how it behaves. Should be used in conjunction to provide full V & V coverage.

21. Types of Testing • Defect Testing • Test cases are designed to expose system defects; For example, • System crashes, incorrect computations • Validation Testing • Intended to show that the software meets its requirements; For example, • Software reliability, performance

22. Key Points • Verification shows conformance with specification. • Validation shows that the program meets the customer’s needs. • V & V is a continuous process which should takes place at each phase of software development life cycle. • V & V techniques involve examination and analysis of the programs for error detection. • Static and dynamic verification techniques should be used in conjunction to provide full V & V coverage.