Verification Planning and Compliance Matrices

1. Verification Planning and Compliance Matrices Brian Selvy Wednesday, August 13, 2014 3:30 – 5:00 p.m. Phoenix West Conference Room

2. Agenda • LSST V&V Process (LSE-160) review with respect to Verification Planning and Compliance Matrices • V&V Implementation methodology in SysML • Overview of Requirements and Verification Planning Spreadsheets

3. References / Support Docs • LSST Verification and Validation Process (LSE-160) – Link to Docushare Released Version • SPIE SysML Implementation Paper – Link • SPIE SysML Implementation Presentation – Link • Requirements eaXL spreadsheet template – Link • Verification Planning eaXL spreadsheet template - Link

4. LSE-160 - LSST V&V Process

5. Verification Planning (Step 3) • Verification Planning is performed on each requirement to define how each will be verified • A verification plan consists of: • Verification Owner: LSST subsystem responsible for verification • Verification Method(s): Test, Demonstration, Analysis, Inspection • Verification Level: verified at Same Level as requirement , or at a Lower Level or Higher Level • Verification Requirement: Statement that defines what will be done to verify the requirement. • Success Criteria: statement that defines explicit pass/fail criteria • Each specific instance of a Verification Method is referred to as a Verification Activity • An individual requirement’s Verification Plan may have more than one Verification Activity

6. Identify Task Interdependency (Step 4) & Schedule Verification Events (Step 5) • Identify Task Interdependency (Step 4) • Some Verification Activities can be naturally grouped and conducted at the same time • These Verification Activities are then grouped into a single Verification Event. • Can result in cost and schedule savings from eliminating redundant or nearly redundant V&V activities • Schedule Verification Events (Step 5) • Events are scheduled, identifying predecessor/ successor relationships and other schedule constraints

7. The Need for Compliance Assessments • In principle, a system’s physical design can be managed two ways: • Design Definition Documents • Formal change controlled documents that define the functional and physical definition of a designed system. They document the design response in the synthesis phase to the provided requirements. • These must be produced and kept up to date for each level of system definition (system, subsystem, element, component, etc.) • Can be cumbersome • Verification Compliance Matrices • Each system provides updated compliance matrices at regular intervals (design reviews, periodically scheduled updates, etc). • Design teams write short statements in response to each requirement to justify if their design complies with each levied requirement or not. • Design teams can reference additional documents with analyses, test results, design details, etc. that substantiate the compliance statements. • Ties in seamlessly with overall Verification Planning activities (I&T planning and Commissioning Planning) • This is the approach LSST is implementing. • Should be less of a burden on teams than the alternative

8. Extending the SysML Stereotype for Verification Planning • SysML does not have a predefined element capable of capturing LSST’s Verification Planning information • SysML is extensible, allowing for the definition of additional stereotypes • LSST created a VerificationPlanning stereotype as an extension of the SysML1.3::requirement metaclass

9. Creation of Verification Plans & Test Cases in the Model

10. Sequencing Test Cases (Verification Events) • Predecessor/ successor relationships • Events that are conducted in parallel/ series • Outside constraints that must be met before a Verification Event can be executed • Results can be used to validate or update the project’s schedule for the Commissioning period. • Test Cases (Verification Events) are sequenced on Activity Diagrams to show:

11. Refinement of Individual Test Cases (Verification Events) • As plans mature, individual Verification Events can be further detailed via association with its own detailed behavior diagram • Serves as refined and more detailed input to the commissioning planning effort • Can be used directly as inputs to writing detailed test & analysis procedures

12. Mapping Individual Test Case Steps to LSST’s PMCS • Refined Test Case Actions mapped to associated Project Management Control System (PMCS) activity steps. • Ensures Verification Activities are included in EVMS

13. V&V Planning & Compliance Spreadsheets • If you do not want to work in the EA SysML directly, you can work in Excel. • LSST has two licenses to an Enterprise Architect add-on called DocX that can be used to create documents from the model and import and export data via Excel. • eaXL is the sub-tool that supports imports and exports from Excel worksheets • Procedures are available for: • Exporting requirements from specification packages • Filling in compliance matrices

14. Extracting Requirements to Excel (1 of 2) • Click on package that contains the requirements you want to extract • Click Extensions -> eaDocX -> eaXL -> Create new workbook

15. Extracting Requirements to Excel (2 of 2) • On Elements tab, click Package and Requirement • On Columns tab, click Stereotype, TV: LSSTRequirements, Name, and Description • Order as shown • Make sure Output Child Elements is clicked • Click “EA “ to export requirements to Excel Workbook • Click File -> Save As to save the file to a desired location. 4 3 2 1

16. Create Verification Package in SysML • Verification Packages Mirror Requirements Specification and ICD Packages • ICD Packages have subsystem and system level verification packages • Allows subsystems to specify which verifications they can do at their level with emulators, simulators, mockups, etc. • Allows PSE to define final interface verification activities during system I&T Requirements Specifications ICDs

17. Create Verification Spreadsheet (1 of 2) • Click on the Verification Package you previously created • Click Extensions -> eaDocX -> eaXL -> Create new workbook

18. Create Verification Spreadsheet (2 of 2) • On Elements tab, click Package and Requirement • On Columns tab, click all of the Tagged Value Types shown • Order as shown • Make sure Output Child Elements is clicked • Click “EA “ to export requirements to Excel Workbook • Click File -> Save As to save the file to a desired location. 4 3 2 1

19. Requirements Spreadsheet Transfer Relevant Data from Req to Verif Spreadsheet • Copy over ElementType and Name in same order to create basic Verification Spreadsheet structure • Replace “requirement” stereotype with “VerificationPlanning” wherever it occurs Verification Spreadsheet

20. Fill out Verif Planning & Compliance • Teams can then fill out all of the Verification Planning and Compliance columns in Excel • When done, please pass the Spreadsheet back to me (Brian Selvy) • I will import back into SysML • I need to ensure Excel row groupings are correct, etc., before importing • I need to check for compliance with items in enumerated lists • I need to check that spreadsheet metadata is correct • I need to ensure that the package is under Version Control so that roll backs are possible if issues arise during import • I will assign unique IDs to all items