Chapter 3 Quality Assurance

1. Chapter 3Quality Assurance SE 471 - Software Testing and Quality Assurance 1

2. Overview • Classification: QA as Dealing with Defects • Defect Prevention • Education and training • Formal method • Other defect prevention techniques • Defect Reduction • Inspection: Direct fault detection and removal • Testing: Failure observation and fault removal • Other techniques and risk identification • Defect Containment • Software fault tolerance • Safety assurance and failure containment 2

3. Quality Assurance from the Correctness Centered Viewpoint (CCV) • According to CCV, Quality means: • few, if any, defects remain in the software system when delivered • the remaining defects cause minimal damage • With the correctness centered quality definition given above, we can view different QA activities as attempting to • prevent, • reduce, or • contain the damage • Briefly discussed here - detailed descriptions are presented in later chapters 3

4. Dealing with Pre-/Post-release Defects • Focus is to remove as many defects as possible before product release - why? • Cost • Reputation • Pre-release QA activities • Defect prevention, detection, removal • Several approaches including development methodologies, programming techniques, defect removing tools, testing, beta tests, etc 4

5. Dealing with Pre-/Post-release Defects • Dormant defects: Faults left in the finished software • May stay dormant for sometime • Potential to cause problem • After release QA activities • Aim at minimizing the negative impact of the remaining faults during operational use after product release • Fixing defects reported by customers • Failure containment: • Most defect containment techniques involve many redundancies • Require significantly more development effort to design/implement • typically limited to the situations where failures are associated with substantial damage 5

6. Graphical Depiction of the Classification Scheme • QA activities act as Barriers - dotted lines • Each barrier • removes or blocks defect sources • prevents undesirable consequences Defect Prevention Activities Defect Removal Activities Fault Tolerance Activities 6

7. Classification of QA Approaches • We can classify these QA alternatives into the following three generic categories: • Defect prevention • Defect reduction • Defect containment 7

8. Defect Prevention • Defect prevention: can be done in two ways: • Eliminating certain error sources. Examples: • Eliminating ambiguities - SRS • Correcting human misconception – lack of knowledge • Fault prevention or blocking missing human actions through the use of certain tools or technologies. Examples: • Missing to put ‘{‘ or ‘(‘ can be avoided by using “Syntax-Directed Editors”. Automatically balances out parenthesis 8

9. Defect Prevention • There are known error sources or missing/ incorrect actions that result in certain fault injections • So? What is needed? • Root cause analysis • Find root causes for injected or potential faults • Error source & Remedy • If human misconceptions are the error sources → education and training • If imprecise designs and implementations that deviate from product specifications → formal methods • If non-conformance to selected processes or standards is the problem → process conformance/standard enforcement • Certain tools or technologies may reduce fault injections 9

10. Defect Prevention • Education & Training • Product and domain specific knowledge • developers unfamiliar with embedded software may design software without considering its environmental constraints - interface problems • General software development knowledge and expertise • lack of expertise with requirement analysis - rework in subsequent design, coding, and testing activities • Knowledge about the specific development methodologies and tools used by the organization • In Cleanroom technology if the developers are not familiar with the key components of formal verification or statistical testing - little chance for producing high-quality products • Knowledge about the software process used by the organization • incremental software development - uncoordinated development may lead to many interface or interaction problems 10

11. Defect Prevention • Formal Method • Formal specification and formal verification • Formal specification is concerned with producing an unambiguous set of product specification • Formal verification checks the conformance of software design or code against these formal specifications • Examples • Axiomatic approach • Planguage, Z notation, VDM • Most influential formal method is the axiomatic approach • Meaning of program statements is abstracted as a set of axioms • Rules are used to compose axioms to build up proofs of entire program correctness • Program behavior is explained as a set of pre-conditions and post-conditions • Obstacle to formal methods: high cost (automated support needed), difficult task (human intensive activities) 11

12. Defect Prevention • Process conformance/standard enforcement may reduce fault injections. Examples: • A better managed process can also eliminate many systematic problems e.g. no process for configuration management may lead to inconsistencies • Lean Software – avoid low quality ‘fat’ software • Certain Tools & Technologies may reduce fault injections. Examples: • Tools, e.g. Syntax directed editor automatically balances out each open parenthesis 12

13. Defect Reduction • Defect prevention activities to be 100% effective – an unrealistic expectation • Need effective techniques to remove as many of the injected faults as possible • Defect reduction: detect and remove faults once they have been injected. • Done in two ways: • Static Analysis: e.g. Inspection of software code, design etc • Dynamic Analysis: e.g. Testing of programs by executing 13

14. Defect Reduction • Inspection • Critical reading and analysis of software code or other software artifacts, such as designs, product specifications, test plans, user manuals etc • Typically conducted by multiple human inspectors • Identified faults need to be removed & removal verified • Used throughout the development process • An economical QA alternative • Two ways to do it: Informal & Formal • Informal reviews: • self conducted reviews/inspections • Formal inspections: • Fagan inspection - most influential work in inspection • Gilb inspection • Inspection processes vary, typically include some planning & follow up activities 14

15. Defect Reduction • Testing • most important parts of QA • the most commonly performed QA activity • Involves execution & observation • When to start • Logically testing can only take place after implementation • However, preparation can start earlier • In the incremental or iterative process, testing can usually get started much earlier • can be divided into various sub-phases starting from the coding phase up to post-release product support, including: unit testing, component testing, integration testing, system testing, acceptance testing, beta testing, etc 15

16. Defect Reduction • Testing • What to test • Black-box (or functional) testing verifies the correct handling of the external functions • White-box (or structural) testing verifies the correct implementation of internal units, structures, and relations among them • White-box test examines source code with focus on: • Control flow • Data flow 16

17. Defect Reduction • Other static & dynamic techniques for defect reduction • Static: formal model based analysis techniques • Algorithm analysis, boundary value analysis, finite state machine, control and data flow analysis, software fault trees etc. • Dynamic: • Simulation and prototyping • Timing and performance analysis for real-time systems • Accident analysis and reconstruction using software fault trees and event trees for safety critical systems • Some Risk identification techniques Will be covered later in detail 17

18. Defect Containment • Defect reduction activities can only reduce the number of faults to a fairly low level, but not completely eliminate them • Still a problem for software systems where failure impact is substantial such as many real-time control software sub-systems used in medical, nuclear, transportation • Few remaining (dormant) faults may be triggered under rare conditions or unusual dynamic scenarios 18

19. Defect Containment • Unrealistic to expect planning of huge number of test cases • Some other means are needed to: • Either contain the damage to local areas so that there is no global failures observable to users • Or limit the damage caused by the software system failure 19

20. Defect Containment • Done in two ways: • Using Fault Tolerance Techniques: • Motivation • Traditional hardware: spare parts and backup units • Fault Tolerance Techniques - Examples • Recovery blocks: rollback and redo • NVP: N-version programming (inspired by hw FT) • Containment Measures: • Avoid catastrophic consequences e.g. death etc • Examples: • Failure containment for real time control software used in nuclear reactors may include concrete walls to encircle and contain radioactive materials in case of reactor melt-down due to software failure 20

21. Defect Containment • Fault Tolerance (FT) VS Exception Handling (EH): • EH deviates from the specification • FT attempts to provide services compliant with the specification after detecting a fault • Important to realize the difference between trying to construct robust software versus trying to construct reliable software • Reliable software by FT or EH? • High profile disaster due to uncaught exception: • On June 4, 1996, the maiden flight of the European Ariane 5 launcher crashed about 40 seconds after takeoff. • The loss was $500,000,000 uninsured • An uncaught exception caused entire software to crash 21

22. Defect Containment • Safety Assurance and Fault Containment • Prevent accidents from happening • accident: failure with severe consequences • In addition to the QA techniques discussed before, various specific techniques are also used for safety critical systems based on analysis of hazards • Hazard elimination: • eliminate hazard sources • Hazard reduction: • reduce the probability of hazard • Hazard control: • reduce the severity of failures • Damage control: • reduce the severity of damage 22

23. Self-Study Assignment – Included in Exam • Read article “Quantifying Quality Requirements Using Planguage”, understand it and practice it for simple examples. Its easy & interesting! • Planguage is derived from Planning & Language. • Designed to quantify qualitative statements in plans, specifications, and designs, Planguage is a keyword-driven language that allows measurable, testable quality requirements to be written. • Planguage has many benefits; it is easy to learn, flexible, compact, extensible, and prevents omissions by providing a consistent set of parameters for quality requirements. • In this article, Planguage keywords and syntax are introduced. Examples of quality requirements before and after using Planguage are given, and the experiences of introducing Planguage are discussed. • You may be given a question in exam/quiz! 23

24. Homework No 2 • Read article “Design by Contract: The Lessons of Ariane” from: http://archive.eiffel.com/doc/manuals/technology/contract/ariane/ • Then read another interesting article which is in fact the critique of the above one titled “Critique of ‘Put it in the contract: The lessons of Ariane’” from: http://home.flash.net/~kennieg/ariane.html 24