SENG 521 Software Reliability & Testing

1. SENG 521Software Reliability & Testing Best Practice SRE (Part 12c) Department of Electrical & Computer Engineering, University of Calgary B.H. Far （far@enel.ucalgary.ca） http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12c/ far@enel.ucalgary.ca

2. Incremental Implementation • Most projects implement the SRE activities incrementally. • A typical implementation sequence: far@enel.ucalgary.ca

3. Feasibility Phase • Activity 1:Define and classify failures • Define failure from customer’s perspective • Group identified failures into a group of severity classes from customer’s perspective • Usually 3-4 classes are sufficient • Activity 2:Identify customer reliability needs • What is the level of reliability that the customer needs? • Who are the rival companies and what are rival products and what is their reliability? • Activity 3:Determine operational profile • Based on the tasks performed and the environmental factors far@enel.ucalgary.ca

4. Feasibility Phase: Benefits • Activity 1 and 2:Define and classify failures, identify customer reliability needs • Benefits:Release software at a time that meets customer reliability needs but is as early and inexpensive as possible • Activity 3:Determine operational profiles • Benefits:Speed up time to market by saving test time, reduce test cost, have a quantitative measure for reliability far@enel.ucalgary.ca

5. Requirements Phase • Activity 4:Conduct trade-off studies • Reliability and functionality • Reliability, cost and delivery date • Activity 5:Set reliability objectives based on • Explicit requirement statements from a request for proposal or standard document • Customer satisfaction with a previous release or similar product • Capabilities of competition • Trade-offs with performance, delivery date and cost • Warranty, technology capabilities far@enel.ucalgary.ca

6. Requirements Phase: Benefits • Activity 4:Conduct trade-off studies • Benefits: Increase market share by providing a software product that matches better to customer needs • Activity 5:Set reliability objectives • Benefits:Release software at a time that meets customer reliability needs but is as early and inexpensive as possible far@enel.ucalgary.ca

7. Design Phase • Activity 6:Allocate reliability among acquired software, components, hardware and other systems • Determine which systems and components are involved and how they affect the overall system reliability • Activity 7:Engineer to meet reliability objectives • Plan using fault tolerance, fault removal and fault avoidance • Activity 8:Focus resources based on operational profile • Operational profile guides the designer to focus on features that are supposed to be more critical far@enel.ucalgary.ca

8. Design Phase : Benefits • Activity 6:Allocate reliability among acquired software, components, hardware and other systems • Benefits:Reduce development time and cost by striking better balance among components • Activity 7:Engineer to meet reliability objectives • Benefits:Reduce development time and cost with better design • Activity 8:Focus resources based on operational profile • Benefits:Speed up time to market by guiding development priorities, reduce development cost far@enel.ucalgary.ca

9. Implementation Phase • Activity 8:Focus resources based on operational profiles • Develop more critical functions first in more detail • Activity 9:Manage fault introduction and propagation • Practicing a development methodology; constructing modular system; employing reuse; conducting inspection and review; controlling change • Activity 10:Measure reliability of acquired software, hardware and other systems • Certification test using reliability demonstration chart far@enel.ucalgary.ca

10. Implementation Phase : Benefits • Activity 8:Focus resources based on operational profile • Benefits:Speed up time to market by guiding development priorities, reduce development cost • Activity 9:Manage fault introduction and propagation • Benefits: Maximize cost-effectiveness of reliability improvement • Activity 10:Measure reliability of acquired software, hardware and other systems • Benefits:Reduce risks to reliability, schedule, cost from unknown software and systems far@enel.ucalgary.ca

11. System Test Phase • Activity 11:Determine operational profile used for testing • Decide upon critical operations • Decide upon need of multiplicity of operational profile • Activity 12:Track testing progress and certify that reliability objectives are met • Conduct feature test, regression test and performance and load test • Conduct reliability growth test far@enel.ucalgary.ca

12. System Test Phase : Benefits • Activity 11:Determine operational profile used for testing • Benefits:Reduce the chance of critical operations going unattended, speed up time to market by saving test time, reduce test cost • Activity 12:Conduct reliability growth testing, track testing progress • Benefits:Know exactly what reliability the customer would experience at different points in time if the software is released at those points far@enel.ucalgary.ca

13. Field Trial Phase • Activity 13:Certify that reliability objectives and release criteria are met • Check accuracy of data collection • Check whether test operational profile reflects field operational profile • Check customer’s definition of failure matches with what was defined for testing the product far@enel.ucalgary.ca

14. Field Trial Phase : Benefits • Activity 13:Certify that reliability objectives are met • Benefits:Release software at a time that meets customer reliability needs but is as early and inexpensive as possible; verify that the customer reliability needs are actually met far@enel.ucalgary.ca

15. Post Delivery Phase /1 • Activity 14:Project post-release staff needs • Customer’s staff for system recovery; supplier’s staff to handle customer-reported failures and to remove faults • Activity 15:Monitor field reliability vs. objectives • Collect post release failure data systematrically • Activity 16:Track customer satisfaction with reliability • Survey product features with a sample customer set far@enel.ucalgary.ca

16. Post Delivery Phase /2 • Activity 17:Time new feature introduction by monitoring reliability • New features bring new defects. Add new features desired by the customers if they can be managed without sacrificing reliability of the whole system • Activity 18:Guide product and process improvement with reliability measures • Root-cause analysis for the faults • Why the fault was not detected earlier in the development phase and what should be done to reduce the probability of introducing similar faults far@enel.ucalgary.ca

17. Post Delivery Phase: Benefits • Activity 14:Project post-release staff needs • Benefits: Reduce post-release costs with better planning • Activity 15-16:Monitor field reliability vs objectives, track customer satisfaction with reliability • Benefits:Maximize likelihood of pleasing customer with reliability far@enel.ucalgary.ca

18. Post Delivery Phase: Benefits • Activity 17:Time new feature introduction by monitoring reliability • Benefits:Ensure that software continues to meet customer reliability needs in the field • Activity 18:Guide product and process improvement with reliability measures • Benefits: Maximize cost-effectiveness of product and process improvements selected far@enel.ucalgary.ca