Software Quality Assurance

1. Software Quality Assurance Economic Evaluation of Software Quality Systems

2. More and more, commercial companies or public organizations are requiring economic evaluation of their quality assurance systems. • Cost of software quality may be viewed as another class of software quality metrics where financial values are used as the measuring tool.

3. Costs of software quality Objectives of cost of software quality metrics

4. In general – it enables management to achieve economic control over SQA activities and outcomes. The specific objectives are: * Control organization-initiated costs to prevent and detect software errors.      * Evaluation of the economic damages of software failures as a basis for revising the SQA budget.      * Evaluation of plans to increase or decrease of SQA activities or to invest in SQA infrastructure on the basis of past economic performance.

5. Managerial control over the cost of software quality is achieved by comparison of actual performance figures with: * Control Budgeted expenditures (for SQA prevention and appraisal activities). * Previous year’s failure costs * Previous project’s quality costs (control costs and failure costs). * Other department’s quality costs (control costs and failure costs).

6. After introducing changes (SQA procedures or infrastructure), the success of the new SQA may be provided by: * Percentage of cost of software quality out of total software development costs. * Percentage of software failure costs out of total software development costs. * Percentage of cost of software quality out of total software maintenance costs. * Percentage of cost of software quality out of total sales of software products and software maintenance.

7. Costs of software quality The classic model of cost of software quality: Feigenbaum’s Model

8. The classic quality cost model, developed in the early 1950s by Feigenbaum and others classifies costs related to product quality into two general classes: ■ Costs of control include costs that are spent to prevent and detect software errors in order to reduce them to an accepted level. ■ Costs of failure of control include costs of failures that occurred because of failure to prevent and detect software errors. Costs of Control Costs of Software Quality Costs of Failure of Control

9. The model further subdivides these classes into subclasses: • ■ Costs of control : • ■ Prevention costs include investments in quality infrastructure and quality activities that are not directed to a specific project or system, being general to the organization. • ■ Appraisal costs include the costs of activities performed for a specific project or software system for the purpose of detecting software errors. They are devoted to detection of software errors. • ■ Costs of failure of control : • ■ Internal failure costs include costs of correcting errors that have been detected by design reviews, software tests and acceptance tests (carried out by the customer) and completed before the software is installed at customer sites. • ■ External failure costs include all costs of correcting failures detected by customers or the maintenance team after the software system has been installed.

10. Prevention costs Feigenbaum’s Model Costs of Control costs Appraisal costs Cost of software quality Internal failure costs Costs of Failure of control costs External failure costs

11. a. Investments in development of SQA infrastructure components * Procedures and work instructions * Support devices: templates, checklists etc * Software configuration management system * Software quality metrics b. Regular implementation of SQA preventive activities: * Instruction of new employees in SQA subjects       * Certification of employees * Consultations on SQA issues to team leaders and others c. Control of the SQA system through performance of: * Internal quality reviews * External quality audits * Management quality reviews

12. (a) Costs of reviews:       *  Formal design reviews (DRs)         *  Peer reviews (inspections and walkthroughs)         * Expert reviews (b) Costs of software testing:         *  Unit, integration and software system tests         * Acceptance tests (carried out by customers)   (c) Costs of assuring quality of external participants by means of design reviews and software testing.

13. * Costs of redesign or design corrections subsequent to design review and test findings * Costs of re-programming or correcting programs in response to test findings * Costs of repeated design review and re- testing (regression tests)

14. Typical external failure costs cover: * Resolution of customer complaints during the warranty period. * Correction of software bugs detected during regular operation. * Correction of software failures after the warranty period is over even if the correction is not covered by the warranty. * Damages paid to customers in case of a severe software failure. * Reimbursement of customer's purchase costs. * Insurance against customer's claims. Typical examples of hidden external failure costs: * Reduction of sales to customers that suffered from software failures. * Severe reduction of sales motivated by the firm's damaged reputation. * Increased investment in sales promotion to counter the effects of past software failures. * Reduced prospects to win a tender or, alternatively, the need to under-price to prevent competitors from winning tenders.

15. Costs of software quality Galin’s extended model

16. Analysis of the software quality costs defined by the classic model reveals that several costs of substantial magnitude are excluded. For example, typical software quality failure costs include: • ■ Damages paid to customers as compensation for late completion of the • project due to unrealistic scheduling. • ■ Damages paid to customers in compensation for late completion of the • project as a result of failure to recruit sufficient staff. • The element common to these two failures is that they result not from any particular action of the development team or any lack of professionalism; they are actually outcomes of managerial failure. • Management can perform several activities to prevent or reduce the costs that result from the types of failure particular to its functions: contract reviews, progress control of the software project.

17. Prevention costs Costs of Control costs Appraisal costs Managerial preparations and control costs Cost of software quality Internal failure costs Costs of Failure of control costs External failure costs Managerial failure costs

18. * Costs of carrying out contract reviews * Costs of preparing project plans, including quality plans * Costs of periodic updating of project and quality plans * Costs of performing regular progress control * Costs of performing regular progress control of external participants’ contributions to projects

19. * Unplanned costs for professional and other resources, resulting from underestimation of the resources in the proposals stage. * Damages paid to customers as compensation for late project completion, a result of the unrealistic schedule in the Company’s proposal. * Damages paid to customers as compensation for late completion of the project, a result of management’s failure to recruit team members. * Domino effect: Damages to other projects planned to be performed by the same teams involved in the delayed projects. The domino effect may induce considerable hidden external failure costs.

20. Costs of software quality Application of a cost of software quality system

21. In order to apply a cost of software quality system in an organization, the following are required: * Definition of a cost of software quality model and specification of cost items: the organization should select the classic or extended model. * Definition of the method of data collection for each cost item: use of Management Information Systems (MIS) in place. * Application of a cost of software quality system, including thorough follow up. * Actions taken in response to the model’s findings.

22. Specification of cost items:

23. Application of a cost of software quality system, including thorough follow up: ■ Assigning responsibility for reporting and collecting quality cost data. ■ Instruction of the team in the logic and procedures of the new system. ■ Follow-up: – Support for solving implementation problems and providing supplementary information when needed – Review of cost reporting, proper classification and recording – Review of the completeness and accuracy of reports by comparing them with records produced by the general MIS system and the cost and activity records from previous periods. This task requires special efforts during the initial implementation period. ■ Updating and revising the definitions of the cost items together with the reporting and collecting methods, based on feedback.

24. Typical actions taken and expected results:

26. General problems * Inaccurate and/or incomplete identification and classification of quality costs. * Negligent reporting by team members * Biased reporting of software costs, especially of “censored” internal and external costs. * Biased recording of external failure costs - “camouflaged” compensation of customers for failures. Problems arising when collecting data on managerial costs: * Contract review and progress control activities are performed in a “part-time mode”. The reporting of time invested is usually inaccurate and often neglected. * Many participants in these activities are senior staff members who are not required to report use of their time resources. * Difficulties in determination of responsibility for schedule failures. * Payment of overt and formal compensation usually occurs quite some time after the project is completed, and much too late for efficient application of the lessons learned.

27. References • Williaw E. Lewis, “Software Testing And Continuous Quality Improvement”, Third Edition, CRC Press, 2009. • K. Naik and P. Tripathy: “Software Testing and Quality Assurance”, Wiley, 2008. • Ian Sommerville, Software Engineering, 8th edition, 2006. • Aditya P. Mathur,“Foundations of Software Testing”, Pearson Education, 2009. • D. Galin, “Software Quality Assurance: From Theory to Implementation”, Pearson Education, 2004 • David Gustafson, “Theory and Problems of Software Engineering”, Schaum’s Outline Series, McGRAW-HILL, 2002.

28. Q & A