Software Quality Assurance

1. Software Quality Assurance Barnali Chakrabarty & Gul Sabah Arif

2. Outline • Software Quality Assurance-Concepts and Misconceptions. • Metrics • SQA in medical device software (MDS). • SQA practices at NASA. • Conclusion.

3. Background and Definitions Software is different from other types of products. • Complexity • Conformity • Changeability • Invisibility

4. Software Quality Engineering • Software Quality Engineering (SQE) is referred to all activities involved in engineering a software product with respect to its quality. This includes activities to achieve quality (Quality Development) as well as to assure the required software quality (Quality Assurance). • Software Quality Assurance (SQA) is referred to as the activities for independent assurance of adherence to defined processed as stated in the CMM key process area on SQA. • Quality Management (QM) is referred to the project intemal activities to monitor and manage the software quality. This includes activities, such as quality planning, process definitions, process monitoring etc • Verification and Validation (V&V) is referred to activities within the project aimed at verifying that the system and its components conforms to their specifications and to validate that the customer needs are fulfilled by the developed product.

5. Software Quality Assurance according to CMM Software quality engineering includes three very basic, but important steps are • Define a software engineering process. • Assure adherence to the process. • Improve the process. • Software process Project paths between process “borders.

6. Process Adherence Activities for process adherence are defined. These are in the CMM referred to as software quality assurance (SQA) activities. The goals for SQA are threefold: • Monitor the software and the development process. • Ensure compliance with standards and procedures. • Bring needs for improvement to managers’ attention.

7. Process improvement • The first step in process improvement is to ensure that the process and its application are conform, i.e. the project execution path falls within the borders, or deviations from the path are under control. • SQA organization • The SQA is an independent observer which shall identify and bring to attention where conformance is lacking between the stated process and the actual execution.

8. SQA vs. quality management • SQA implementation

9. Metrics In general, for most software quality assurance systems the common software metrics that are checked for improvement are:- • the Source lines of code • cyclical complexity of the code • Function point analysis • bugs per line of code • code coverage • number of classes and interfaces • cohesion and coupling between the modules

10. SQA standards used in Medical Device Software (MDS) • The proliferation of medical device software (MDS) potentially increases the risks of patient injury from software defects. • Software quality assurance problems are a major cause of the defects behind such faults. The Center for Devices and Radiological Health (CDRH) of the U.S. Food and Drug administration (FDA) reviewed medical device recalls due to software failures from 1983 to 1991 and estimated that 90% were due to inadequate design and 19% were due to inadequate change control. An individual problem in software quality assurance or software process control (SQA) can cause defects in many products, and many of the defects are difficult to identify. • The Food and Drug Administration (FDA) in 1998 updated its MDS regulations and software design and development has been formally included under the Quality Systems Regulation (QSR), as outlined in the Code of Federal Regulations (CFR) . The QSR is a federal system of regulations for good manufacturing practices for drugs and medical devices.

11. SOFTWARE DEFECT-RELATED FAULTS INITIATING RECALLS OF COMPUTERIZED MEDICAL DEVICES

12. Federal regulations for MDS • Federal regulations for medical devices must cover a wide range of contrivances . Scissors and stand-alone software are both medical devices, and the regulatory approach toward computer- controlled devices has been to include them within existing regulations through modifications, amendments, or supplemental guidance documents. • The FDA does not want to require or review what may be unnecessary, expensive, or burdensome administrative paperwork. • One approach involves the use of third-party SQA standards. Major sections of the QSR and FDA guidelines for application to software development parallel general aspects of third-party SQA standards. • FDA has not specified whether third-party SQA standards by themselves can be used to meet federal requirements for modifications of existing MDS, and such standards and certifications are not included in the FDA’s list of recognized consensus standards. • The purpose of is to develop a framework to assess whether third-party SQA standards meet FDA requirements.

13. A BRIEF OVERVIEW OF ISO 9000-3 More than 25 000 companies have registered with the ISO and more than 1800 companies and 90 countries have effected ISO standards. • CMM Levels 2 and 3 are closely comparable to ISO 9001 and suppliers certified at either level should have little difficulty obtaining ISO 9001 certification. • METHODOLOGY Specifications from the QSR and FDA guidelines were identified and then closely reviewed by the research team for both general and specific requirements or guidelines pertaining to the development of MDS. An initial assessment was then made of each of the resultant sets of categorical specifications with the ISO and CMM standards. SQA standards and corresponding federal regulations in the framework were then cross-referenced.

14. ASSESSMENT FRAMEWORK • The assessment framework consisted of four major areas of practice derived from the FDA requirements: • Process management. • Organizational Quality System. • Software Configuration Management. • Software Project and Development Planning. • Verification and Validation Planning • Requirements specification. Requirements specification is the identification of the requirements of the software, the system, and the user that will determine the specifications used to develop software. • Design control. Design control is the control of device design to ensure that specified design requirements are met. • Change control. Change control is defined as the handling of software that does not conform to specified requirements and the implementation and tracking of the subsequent software change process.

15. ASSESSMENT FRAMEWORK Process management. Process management is defined here to include the management of the organizational quality system, the project, the development process, the individual software items underdevelopment, and the processes of assuring that software items meet specifications. 1.Organizational Quality System. The organizational quality system is the establishment and implementation of quality policy,systems, and practices across an organization. 2.Software Configuration Management. Software configuration management (SCM) refers to the identification of software items and the documented tracking of them throughout the software lifecycle. 3.Software Project and Development Planning. Software project and development planning is the implementation and of the control planning and development processes. • Verification and Validation Planning. Verification and validation planning is the establishment and implementation of procedures to continuously ensure that software items meet the specified requirements.

16. RESULTS

17. CONCLUSION • ISO 9000-3 guidelines for application of ISO 9001 closely match the requirements for quality systems management but are not as extensive as the CMM in most software- specific areas and have more specific omissions in other areas. The CMM meets or exceeds most software-specific FDA requirements but may require adaptation to meet the requirements for general quality systems. Neither standard meets every FDA requirement. • It has identified the potential strengths and weaknesses of the ISO and CMM standards in the specific context of medical device software. • It has demonstrated evidence that both standards can make significant contributions toward enabling suppliers to meet FDA requirements. • It has also identified aspects of the SQA standards that may need modification in order to meet FDA requirements, which may be useful to suppliers and third-party SQA standard organizations alike.

18. Software Quality Assurance Engineering At NASA

19. Introduction At NASA , the criteria for the evaluation of software is done using the McCall Software Quality Criteria

20. Here are the list of definitions for quality criteria for software • Correctness • Efficiency • Flexibility • Integrity • Interoperability • Maintainability • Portability • Reliability • Reusability • Testability • Usability

21. Software Quality assurance requires both process assurance and Product Assurance • Process Assurance-provides management with object feedback regarding process compliance to approved plans,procedures and standards. • Product Assurance-focuses on the quality of the product within each phase of life cycle such as the requirements,design,code and test plan

22. Process Assurance at nasa NASA, in the past have developed their own standards for product assurance but recently have recognized the value of using commercial standards . Some of the most common software development models for process assurance are:- • Capability Maturity Model that has recently developed into the Capability Maturity Model Integrated • SPICE • IEEE Standard for developing software lifecycle processes

23. Product Assurance AT NAsA At NASA’s Goddard Space Flight Centre ,product assurance is assigned to the Office of System Safety and Mission Assurance. It is carried out by an independent group of people whose function is to monitor the implementation of quality. The assurance management have created a list of tasks that SQA should perform at each phase of the lifecycle.

24. Some of the activities within the phases are listed as follows: • Concept Phase Activities • generate or assist in the generation of concepts of operations • identification of project risks and mitigation strategies • development and review of various programs • facilitate resolution of issues, concerns and risks

25. Requirements Phase Activities • Generate or assist in the generation of requirements • Analyze and review requirements for industry • Execute Automated Requirement Measurement tool and resolve any requirements quality related issues • Assist in tracking or facilitation of problem reports

26. Design Phase Activities • Attend or participate in design reviews and track or maintain any issues, tracking logs and tools • Review and analyze design for characteristics that are required and industry acceptable • Review or track project metrics for trends, risks and potential problems

27. Implementation Phase Activities • Attend code walkthroughs and peer reviews. Participate in the tracking and resolution of any issues,etc • Review and /or access code per organization's coding standards . • Generate and/or review requirement to code traceability/verification matrices • Review or track program or project for trends, risks and potential problems

28. Test phase Activities • Generate and /or review test defects/test reports and other test related artifacts as applicable • Generate and/or review requirement to code traceability/verification matrices • Attend and /or participate in test readiness, operational readiness and launch readiness reviews • Attend change control and defect review board meetings and participate in the assessment of changes and/or defects

29. Operations and Maintenance phase Activities • Generate post mortem reports and lessons learned • Attend change control and defect review board meetings and participate in the assessment of changes and/or defects

30. Some of the activities that are performed throughout all lifecycle phases • Conduct audits of program/project processes and/or products • Generate lessons learned throughout lifecycle phases and review these lessons for applicability to projects

31. Safety Critical Software According NASA Software Safety Standard • Software that directly ,or indirectly ,contributes to the occurrence of a hazardous system state ,controls or monitors safety critical functions ,runs on the same state as safety critical software or impacts systems which run safety critical software, or handles safety critical data

32. Reliability • At NASA,satelites fly for multiple years, thus associated software must be reliable to support the expected lifetime of the satellite.

33. Independent Verification and Validation (IV &V) • At NASA,Software IV & V is defined as a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle. • NASA has a facility in West Virginia whose primary purpose is the accomplishment IV & V.This Facility is part of GSFC’s Office of Systems Safety and Mission Assurance ,recognizing relationship and importance of IV & V to SQA

34. Conclusion • Software Quality Assurance is faced with many challenges starting with the method of defining quality for software. There needs to be a common understanding as to what is high quality software, but the final definition is usually influenced by the environment of the software usage

35. References 1: A framework for assessing the use of third-party software quality assurance standards to meet FDA medical device software process control guideline'sBovee, M.W.; Paul, D.L.; Nelson, K.M.;Engineering Management, IEEE Transactions onVolume 48,  Issue 4,  Nov. 2001 Page(s):465 – 478  2.FDA: Between Process and Product EvaluationAbdeen, Marwan M.   Kahl, Wolfram   Maibaum, Tom   This paper appears in: High Confidence Medical Devices, Software, and Systems and Medical Device Plug-and-Play Interoperability, 2007. HCMDSS-MDPnP. Joint Workshop onPublication Date: 25-27 June 2007 page(s): 181-186  3.   Software quality assurance-concepts and misconceptionsRuneson, P.; Isacsson, P.;Euromicro Conference, 1998. Proceedings. 24thVolume 2,  25-27 Aug. 1998 Page(s) :853-859 4. Software quality assurance engineering at NASARosenberg, L.H.; Gallo, A.M., Jr.;Aerospace Conference Proceedings, 2002. IEEEVolume 5,  2002 Page(s):5-2569 - 5-2575 vol.5