SENG 521 Software Reliability & Testing

1. SENG 521Software Reliability & Testing Cleanroom Development (Part 12a) Department of Electrical & Computer Engineering, University of Calgary B.H. Far （far@enel.ucalgary.ca） http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12a/ far@enel.ucalgary.ca

2. What is it ? /1 • A software engineering methodology with the emphasis on design with no defects and test based on statistical quality control. • Analysis and design models are based on incremental software model and created using box structure representation. • A “box” encapsulates the system (or some aspect of the system) at a specific level of abstraction. • Correctness verification is applied once the box structure design is complete. far@enel.ucalgary.ca

3. What is it ? /2 • The software is tested by defining a set of usage scenarios (i.e., operational modes), determining the probability of use for each scenario (i.e., operational profile), and then defining random tests that conform to the probabilities. • The error records are checked. No corrective actions are taken. Only certification test is conducted to check whether errors (i.e., failure intensity) meet the projected reliability for the software component. far@enel.ucalgary.ca

4. Increment 1 RG RG BSS BSS FD FD CV CV CG CG CI CI SUT SUT C C TP TP Increment 2 Cleanroom Process Model SE CG: Code Generation CI: Code inspection SUT: Statistical use testing C: Certification test TP: Test planning SE: System engineering RG: Requirement gathering BSS: Box structure specification FD: Formal design CV: Correctness verification far@enel.ucalgary.ca

5. Cleanroom Strategy /1 • Requirement gathering (RG) • A detailed description of customer level requirements for each increment. • Box structure specification (BSS) • Functional specification using box structure to separate behavior, data and procedures. • Formal design (FD) • Specifications (black boxes) are refined to become analogous to architectural (state boxes) and procedural (clear boxes) design. far@enel.ucalgary.ca

6. Cleanroom Strategy /2 • Correctness verification (CV) • A set of correctness verification activities on the design and moves later to code. First level verification is via application of a set of “correctness questions”. • Code generation, inspection & verification (CG & CI) • The box structure transformed to a programming language. Walkthrough and code inspection techniques are used to ensure semantic conformance with the box structure. far@enel.ucalgary.ca

7. Cleanroom Strategy /3 • Statistical test planning (TP) • Planning the test based on operational modes, operational profiles and reliability. • Statistical use testing (SUT) • Creating test case, execute them and collecting error data. • Certification (C) • Conducting certification test rather than reliability growth to accept/reject developed software components (using reliability demo chart, etc). far@enel.ucalgary.ca

8. Box Structure /1 • Black box • Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response. • State box • Encapsulates state data and services (operations). Input to the state box and outputs are represented. • Clear box • Transition function that are implied by the state box. It contains the procedural design of the state box. far@enel.ucalgary.ca

9. CB1.1.1 BB1.1.1 SB1.1.1 CB1.1.2 BB1.1 BB1.1.2 CB1.1.3 BB1.2 BB1 BB1.1.3 BB1.n Box Structure /2 clear box State box Black box far@enel.ucalgary.ca

10. State T f: S* → R S R S R f: S* → R Black box State T State box g12 R S g11 cg1 g13 clear box Box Structure /3 far@enel.ucalgary.ca

11. Design Refinement • If a function f is expanded into a sequence g and h, the correctness rule for all input to f is: • Does g followed by h do f? • If a function f is expanded into a condition if-then-else, the correctness rule for all input to f is: • Whenever condition <c> is true does g do f and whenever <c> is false does h do f? • When function f is refined as a loop, the correctness rule for all input to f is: • Is termination guaranteed? • Whenever <c> is true, does g followed by f do f? and whenever <c> is false, does skipping the loop still do f? far@enel.ucalgary.ca

12. Advantages of Verification • Design verification has the following advantages: • Verification is reduced to a finite process. • Every step of design and every line of code can be verified. • Near zero defect level is achieved. • Scalability is possible. • Better code than unit testing can be generated. far@enel.ucalgary.ca

13. Cleanroom Testing • Using “statistical use testing”. • Determine a “usage probability distribution” via the following steps: • Analyze the specification to identify a set of stimuli (direct and indirect input variables). • Create usage scenarios (operational modes). • Assign probability to use of each stimuli (operational profile). • Generate test cases for each stimuli according to the usage probability distribution. far@enel.ucalgary.ca

14. Certification Test • Cleanroom approach DOES NOT emphasize on • Unit or integration testing. • Bug fixing as a result of test and regression. • Certification procedure involves the followings: • Create usage scenarios. • Specify a usage profile. • Generate test cases from the profile. • Execute test cases and record failure data. • Compute reliability and certify the component or system using reliability demo chart, etc. far@enel.ucalgary.ca

15. Reliability Demo Chart /1 • An efficient way of checking whether the failure intensity objective (λF) is met or not based on collecting failure data at time points. • Vertical axis: failure number • Horizontal axis: normalized failure data, i.e., failure time x λF far@enel.ucalgary.ca

16. Certification Example far@enel.ucalgary.ca

17. Conclusions • Cleanroom approach is a rigorous approach to software engineering that has emphasis on: • Mathematical verification of correctness of design. • Certification of software reliability. • Cleanroom approach is yet to become a common practice in software development industry because of emphasis on the above two points. far@enel.ucalgary.ca