Breakout Group 2: Software Quality Assurance

1. Breakout Group 2:Software Quality Assurance Outcome

2. Objectives • Review 2009 Workshop Outcome. • Three talks on SW QA activities at accelerator labs. • Paul Wright – SNS • EnzoCarrone – SLAC • Kelly Mahoney - JLab • Identify SW Assurance activitiesfor high impact operations at Accelerator Facilities. • Recommendations for incorporating value added SW Assurance activities into a revised ASO guidance document.

3. Alignment with the Safety Order DOE Responsibilities • Facility operations meet DOE mission and operational objectives • Operations comply with safety program and objectives • Ensure facility safety program incorporates: • ASE and SAD • clearly defined roles and responsibilities • a configuration management process • readiness review • inventory of exempt accelerators Major Constituents of the CRD • Accelerator Safety Envelope (ASE) • Bounding conditions for safe operations • Safety Assessment Document (SAD) • Describe Engineered and Administrative Controls • Unidentified Safety Issue (USI) • Configuration Management • Accelerator Readiness Review (ARR) • Contractor Assurance Process • Configuration Management Program • Administrative Processes Related to Accelerator Safety

4. Paul Wright – SNS • SNS is moving to “One Box” two programmer safety PLC system. • Using ISO 13849-1:2006 safety of machinery as the best fit for interlock systems. This standard also addresses software QA requirements. • Implementation of best practices • Expanded code documentation • Trace Safety Functions from requirements to implementation • Making more use of certified safety functions (code) in PLCs • Question – should one trade the two-programmer model for adoption of predefined safety functions? • Is there benefit to moving to a third programmer that is trained as an independent technical reviewer of safety code. • Emphasis in testing is moving away from functions defined in the requirements (success path) to unusual event case. Justification is that the success path is effectively tested each time someone walks through a door. Rare cases may hide systemic faults.

5. EnzoCarrone – SLAC • SLAC is moving to a distributed networked safety PLC architecture. • A Safety Integrity Level (SIL) based process provides a better measure of system effectiveness than a system where safety layers are piled on. • In order to benefit from an SIL model, one must know the reliability information for not only the PLCs, but also the field devices such as relays. In SLAC’s case, some of the field devices are 40 year old. • Full compliance with an industry standard like ANSI/ISA S84 is difficult to achieve. If one uses the standard as best practice guidance, where is the line between must and should in the application of the standard. • SLAC Safety Section is moving to a service provide model. A standard work process is developed and used in multiple projects. Software QA is deeply integrated in to the model. • SLAC is modeling requirements in MatLab/Simulink before coding begins.

6. Kelly Mahoney – JLab • JLab is implementing a software QA procedure that is based on risk • Applies to all software designed or modified at the Lab • Compliments cyber security process • Risk assessment method is applied to six areas of potential impact to safety, mission, and goals. • Provides the tools and guidance necessary to apply it throughout the Lab • Incorporates metrics and continuous improvement process

7. Emergent Themes • How much Software QA is enough? Many Labs face an onerous process for small changes that does not appear to be proportional to the risk. • How much cyber security is enough? Even with an air gap to the outside world, programming PCs need to be updated and maintained. Malware is now transferred through portable media such as USB sticks. • What are appropriate qualifications for a programmer of safety software? • Need to address the requirements of one-box safety PLC solutions. • Labs are moving toward automated testing and targeted testing, e.g. test vectors. • It is still difficult to determine reliability of legacy or in-house designed safety system components.

8. Outcomes: • Software QA (Assurance) needs to be emphasized in a revised ASO guidance document. • Community should clarify methods and requirements of 414.1C part 5 in the context of accelerator safety. • Continue to develop model based requirements and specifications. • Continue to pursue simulator based bench testing for software. • With simulation and model based testing to build confidence in the software, it may be possible to both reduce the field test time and extend certification intervals to 2 years or more while reducing spurious trip rate. • Group 2 will remain in contact to continue the discussion on these topics.

9. Places where better ASO guidance will help: • Clearly defined Roles and responsibilities • What is an acceptable level of qualification for a safety system programmer? Designer? • What is an appropriate level of independence between the system designers, reviewers, and testers. • Configuration Management Process • Software management process needs to be proportional to the risk mitigated by the safety function. • SW Management • Cyber Security Management • Requirements Management • Tracability from design basis, e.g. SAD, to installed system • Bounding conditions for safe operations • Impact of system failures attributable to software • Impact of cyber security failures to system integrity • Describe Engineered and Administrative Controls • Describe SWQA process attributes as applied to safety controls • Safety PLC software • Software maintenance and versioning • Software assurance of design basis simulation and modeling codes • Appropriate use of a “One Box” safety system • Configuration Management Program • SW management systems • Physical security management • Cyber security management • Administrative processes Related to Safety • CMS • Continuous improvement • Software Test and Management processes • Clarify use of 414.C(D) in accelerator safety