1 / 31

Software Test and Verification (T&V) Process Training

Software Test and Verification (T&V) Process Training. Process for planning and executing software testing and verification. Agenda. Software Processes. The SEPG has developed the following processes: GRC-SW-7150.3 - Software Project Planning

ziarre
Download Presentation

Software Test and Verification (T&V) Process Training

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Software Test and Verification (T&V) Process Training Process for planning and executing software testing and verification

  2. Agenda

  3. Software Processes • The SEPG has developed the following processes: • GRC-SW-7150.3 - Software Project Planning • GRC-SW-7150.4 - Software Project Monitoring and Control • GRC-SW-7150.5 - Requirements Development • GRC-SW-7150.6 - Requirements Management • GRC-SW-7150.8 – Software Measurement • GRC-SW-7150.9 - Software Configuration Management • GRC-SW-7150.10 - Transition SW to a Higher Class • GRC-SW-7150.12 - Formal Inspection and Peer Review • GRC-SW-7150.14 - Software Acquisition SOW Guideline (being updated) • GRC-SW-7150.15 - Software Acquisition Planning • GRC-SW-7150.16 - Software Estimating • GRC-SW-7150.17 - Software Safety Planning • GRC-SW-7150.18 - Software Test and Verification • These are available on Software@Glenn.

  4. Software Documentation Templates • The SEPG has developed the following document templates: • GRC-SW-TPLT-SMP - Software Management Plan Template • GRC-SW-TPLT-SRS - Software Requirements Specification Template • GRC-SW-TPLT-RTM - Requirements Traceability Matrix Template • GRC-SW-TPLT-SCMP - Software Configuration Management Plan Template • Software Change Request/Problem Report • GRC-SW-TPLT-MMS – Master Metrics Spreadsheet • GRC-SW-TPLT-SDD - Software Design Description Template • GRC-SW-TPLT-IDD - Interface Design Description Template • GRC-SW-TPLT-STP - Software Test Plan Template • GRC-SW-TPLT-STPr - Software Test Procedure Template • GRC-SW-TPLT-STR - Software Test Report Template • GRC-SW-TPLT-SVD - Software Version Description Template • GRC-SW-TPLT-SUM - Software Users Manual Template • GRC-SW-TPLT-SMntP - Software Maintenance Plan Template • GRC-SW-TPLT-SSP - Software Safety Plan Template • GRC-SW-TPLT-SSCA - Software Safety Criticality Assessment (new) • These are available on Software@Glenn. • They are Word documents except for the Master Metrics SS.

  5. Test and Verification Process • This process was developed to provide software developers with a way to plan and implement the test and verification activities of the software for a project. • Aids in planning the testing activities at the non-verification level including unit testing, and project support testing • Aids in planning and implementing the formal verification of the software for the project • A note about Validation: Validation is usually performed at the system level, and as such would follow the project Verification and Validation Plan. However, if there is no project V&V Plan, or if any software-level validation testing is required, it should be included as part of the software test planning effort. • This process is used throughout the software development project, beginning with the planning of the overall software effort through the conclusion of the formal verification and validation. • Implements NPR 7150.2B SWE-146, SWE-28, SWE-29, SWE-30, SWE-31, SWE-62, SWE-65, SWE-66, SWE-67, SWE-68, SWE-69, SWE-71, SWE-72, SWE-87, (SWE-39 if acquisition) • While we are not currently tracing to the verification process area in CMMI, we will eventually add that traceability information to the document.

  6. Test and Verification Process

  7. Test and Verification Process Identify approach to software testing(6.1) This step is typically performed in the planning phase of the project lifecycle. The Software T&V Lead along with the Software Project Lead identifies the approach to software testing. Items to consider are: • a) Overall approach to software testing (e.g. procedure-based, automated) • b) At what levels the software will be tested (CSU, CSC, CSCI, System) • c) What type of testing will be performed (e.g. white box, functional, integration, verification) • d) Special tests (e.g. timing, capacity, storage, processor utilization) • e) Critical interface tests. How early? How often? • f) Off-nominal tests • g) Approach for regression testing • h) Plan for data recording and analysis • i) Hardware and software fidelity for each type of test. • j) Simulators/emulators needed • k) Amount of code coverage (portion of code tested) • l) Software-level validation testing (usually not applicable) • Note: Even though the software test approach should include requirements verifications to be performed by test, it is not synonymous with the software verification approach. The test approach includes more than just verification testing. Likewise, the verification approach should include all verification methods, not just test.

  8. Test and Verification Process Identify software test support (6.2) • This step is typically performed in the planning phase of the project lifecycle. The Software Test Lead along with the Software Project Lead works with the other subsystem/discipline leads to determine any software needed to support any subsystem or component development or checkout tests. This is especially important for early avionics hardware testing. • The Software Test Lead works with the project Lead Systems Engineer or Assembly, Integration and Test Lead to identify environmental, assembly, and system checkout tests that software will need to support. Do not forget to include support for system level Validation testing. Note: Remember to think about simulators and emulators that either the software team will need for development and test, or the other subsystems might need to check out their products.

  9. Test and Verification Process Identify Software Verification Approach (6.3) • This step is typically performed in the planning phase of the project lifecycle. The Software Test Lead along with the Software Project Lead determines the methods of verification to be used, as well as any special approach to verification of the software requirements. The methods typically used are Test, Analysis, Inspection, and Demonstration. Consider how Software Quality Assurance will be involved as well as how any issues found during verification will be documented and tracked to closure.

  10. Test and Verification Process Identify verification methods for the software requirements (6.4) • The Software Test Lead or designated assignee performs analysis on the software requirements to determine the appropriate method(s) of verification. • This information is documented in the Software Requirements Specification (per GRC-SW-7150.5), in requirements verification statements, the requirements traceability matrix or in the Software T&V Plan. Note: the verification work products are determined in this step.

  11. Test and Verification Process Determine Test Schedule (6.5) • The Software Test Lead and Team review the tests planned and determine when the software will be available. The test lead analyzes the availability of resources and develops a testing schedule. The Software Lead works with the project to merge the software test schedule with the overall project test schedule

  12. Test and Verification Document Test & Verification Plan (6.6) • The Software Test Lead combines all of the compiled information into the Software Test and Verification Plan. Note: The Software Test Plan (GRC-SW-TPLT-STP) can be used for this document. For smaller project, the information can be documented in the Software Management and Development Plan (SMDP). The verification information can also be documented in the overall project V&V Plan.

  13. Test and Verification Conduct Review/Inspection of the SW T&V Plan (6.7) • The Software Test Lead provides the SW T&V Plan to the project for review or through a formal inspection process per GRC-SW-7150.12, Formal Inspection and Peer Review Process. Note: the project review could be as informal as a peer review or ERB, or formal in a milestone review.

  14. Test and Verification Perform Non-Verification Testing (6.8) • The software team lead and team members perform non-verification testing and analysis as documented in the SW T&V Plan. This level of testing is used to check the code throughout development and in preparation for formal verification.

  15. Test and Verification Develop Formal Verification Procedures (6.9) • The Software Test Lead develops the formal verification procedures for the software products based on the baselined software requirements and operational concepts. The GRC-SW-TPLT-STPr, Software Test Procedure, Template can be used to document the procedures. • Software Test Lead performs dry-runs of the procedures to ensure completeness. Note: When developing the test procedures, include a section for test failure management that explains how the team will handle failures/anomalies (see Saffire 1-2-3 T&V Plan section 4.7 for an example).

  16. Test and Verification Conduct Review/Inspection of Procedures (6.10) • The Software Test Lead holds a review or formal inspection of the verification procedures to ensure completeness and correctness, per GRC-SW-7150.12, Formal Inspection and Peer Review Process. Note: the type of review would be up to the project. The project may want to conduct an ERB or formal review in place of an inspection.

  17. Test and Verification Obtain project signatures (6.11) • The Software Test Lead provides the updated procedures to the project for review and signature, if applicable. Note: The project typically determines who needs to sign off on the test procedures. At a minimum, the Software Lead and the Software Assurance Lead are recommended. For smaller projects, this step may not be required.

  18. Test and Verification Conduct Verification Readiness Review (VRR) (6.12) • The Software Lead performs a review to determine if everything is ready for the software verification. Note: The project size and resources should determine the level of formality and size of the review. It could range from a formal Verification Readiness Review to a simple peer review. Ensure that the software team is included in the review, at a minimum.

  19. Test and Verification Conduct Formal Verification (6.13) • The Software Test Lead along with members of the Software Team and the Software Quality Assurance Team perform the formal verification of the software. • If anomalies are encountered during the execution of the tests or afterwards when results are analyzed, they are documented appropriately and the software test lead and team continue to step 6.13.1, otherwise they move on to step 6.14.

  20. Test and Verification Analyze Anomalies (6.13.1) • At a minimum, the software T&V lead and test team review any anomalies that were found and determines the appropriate corrective action. Additional testing to determine the root cause or to re-create the anomaly may be needed. • The resulting analysis is reviewed with the Software Project Lead and Software Quality Assurance. The results of the analysis are provided to the Project Manager as necessary. The project may decide to issue a deviation or waiver to the requirement. • If a change is needed, proceed to step 6.13.2, otherwise proceed to step 6.14.

  21. Test and Verification Update Software or Test Procedure (6.13.2) • The software team performs the corrective actions determined during the analysis performed in step 6.13.1. The corrective action could be to update the software, or it could be an update to the verification procedure following the project’s change process. • Return to step 6.12 to re-verify the software using the Formal Verification Procedures. Note: There may not be any corrective action if the team decides to leave the anomaly because it will not impact the mission success. All corrective action documentation is taken care of in this step.

  22. Test and Verification Write Verification Report(s) (6.14) The Software Test Lead gathers all the As-Run Procedures and writes verification reports. Verification reports should include: • Brief Description of the Verification • Date Verification Performed • Anomalies if any occurred and associated corrective actions • Results of the final verification • The As-Run Procedure(s) • Any deviations or waivers The test lead provides this report to the Software Project Lead and Software Quality Assurance for approval. Note: the GRC-SW-TPLT-STR, Software Test Report, template can be used to document the verification report information. Note: A verification report is usually only written after the final successful test concludes, however, information about any previous unsuccessful tests should be included in the final report and all documentation associated with the verification should be archived in step 6.14. Note: All As-Run procedures should be archived, but only the final fully successful one should be included with the verification report.

  23. Test and Verification Close Anomaly Reports and Archive Verification Products (6.15) • The Software Test Lead ensures that all anomaly reports associated with formal verification are closed and gathers all documentation associated with verification for archival. • All collected documentation is provided to the Project Configuration Management (CM) person for storage and project archival.

  24. Test and Verification Metrics and Reporting • Metrics • Refer to GRC-SW-7150.8 ‘Software Measurement’ for project metrics and the process for collecting and reporting metrics. • Reporting • For all verifications, anomalies and test results need to be reported. Anomalies should be reported immediately and the software team should follow the process the project requests for resolving them. At the conclusion of each verification test, the results should be reported to the project.

  25. Software Test and Verification ProcessResources, Tools, and Templates • Available from Software@Glenn: • http://software.grc.nasa.gov • GRC-SW-7150.18: Software Test and Verification Process • NPR 7150.2B: NASA Software Engineering Requirements • GRC-SW-TPLT-STP: Software Test PlanTemplate • GRC-SW-TPLT-STPr: Software Test Procedure Template • GRC-SW-TPLT-STR – Software Test Report • Available from the Agency • NASA Engineering Network: Software Engineering Community of Practice • https://nen.nasa.gov/web/nen/home: Click on Communities->Software Engineering • NASA Software Engineering Handbook • https://swehb.nasa.gov/

  26. Feedback on the Test and Verification Process • Processes, checklists, templates and forms available at Software@Glenn: http://software.grc.nasa.gov • To ask questions • Lisa Lambert, x3994 • grc-sepg-lead@lists.nasa.gov • We value your feedback • The feedback form is on the Feedback page of Software@Glenn. • Send the feedback form to grc-sepg-lead@lists.nasa.gov. • Group feedback sessions available upon request. • Based on feedback, the process will be updated. • The updated process will be made available on Software@Glenn. • Share your products as examples for future projects!

  27. Questions?

  28. Backup

  29. NPR 7150.2B Requirements SWEs: SWE-146, SWE-28, SWE-29, SWE-30, SWE-31, SWE-62, SWE-65 • SWE-146: The project manager shall define the approach to the automatic generation of software source code including: a. Validation and verification of auto-generation tools. b. Configuration management of the auto-generation tools and associated data. c. Identification of the allowable scope for the use of auto-generated software. d. Verification and validation of auto-generated source code. e. Monitoring the actual use of auto-generated source code compared to the planned use. f. Policies and procedures for making manual changes to auto-generated source code. g. Configuration management of the input to the auto-generation tool, the output of the auto-generation tool, and modifications made to the output of the auto-generation tools. • SWE-028: The project manager shall plan software verification activities, methods, environments, and criteria for the project. • SWE-029: The project manager shall plan the software validation activities, methods, environments, and criteria for the project. • SWE-030: The project manager shall record, address, and track to closure the results of software verification activities. • SWE-031: The project manager shall record, address, and track to closure the results of software validation activities. • SWE-062: The project manager shall unit test the software code per the plans for software testing. • SWE-065: The project manager shall establish and maintain: a. Software test plan(s). b. Software test procedure(s). c. Software test report(s).

  30. NPR 7150.2B Requirements(cont.)SWEs: SWE-66, SWE-68, SWE-69, SWE-71, SWE-72, SWE-87, SWE-55, (SWE-39, SWE-42 if acquisition) • SWE-66: The project manager shall perform software testing. • SWE-67:The project manager shall verify the requirement to the implementation of each software requirement. • SWE-68: The project manager shall evaluate test results and record the evaluation. • SWE-69: The project manager shall record defects identified during testing and track to closure. • SWE-71: The project manager shall update software test plan(s) and software test procedure(s) to be consistent with software requirements • SWE-72: The project manager shall provide and maintain bidirectional traceability from the software test procedures to the software requirements. • SWE-87: The project manager shall perform and report the results of software peer reviews or software inspections for: a. Software requirements. b. Software plans. c. Any design items that the project identified for software peer review or software inspections according to the software development plans. d. Software code as defined in the software and or project plans. e. Software test procedures.

  31. NPR 7150.2B Requirements(cont.)SWEs: SWE-39 for acquisition • SWE-39: The project manager shall require the software supplier(s) to provide insight into software development and test activities; at a minimum, the software supplier(s) will be required to allow the project manager or designate to: a. Monitor product integration. b. Review the verification activities to ensure adequacy. c. Review trades studies and source data. d. Audit the software development process. e. Participate in software reviews and systems and software technical interchange meetings.

More Related