IV&V Testing Strategies for Independent Verification of NASA Mission Software Implementation

1. IV&V Testing Strategies for Independent Verification of NASA Mission Software Implementation 3rd Annual Workshop on Independent Validation and Verification September 17, 2009

2. Objectives • Vision for an IV&V Test Environment • Value of an IV&V Test Environment • IV&V Test Environment Concept of Operations • Characteristics of an IV&V test environment • Software IV&V Testing Strategies • Current status of the IV&V Test Environment

3. Vision: Problem Statement • The current method of analyzing source code in its static state doesn’t adequately evaluate the behavior of the code under operational and environmental conditions • System interactions, dependability, quality attributes, among other properties are hard to evaluate with static analysis

4. Vision: Scope • Develop, maintain and operate an adaptable test environment for the IV&V Program that enables the dynamic analysis of software behavior for multiple NASA missions • Compliment the inspection of the development test design and subsequent test results as well as manual and automatic source code analysis in its static state • These methods have been effective at uncovering evidence that the implementation answers the three questions

5. Vision: Value • Uncover issues that would not be found by static code analysis, manual requirements based code inspection, or test results inspection • Allow for the creation of goodness statements that will add more value to the project and provide a greater confidence in the assurance of the software

6. Concept of Operations • Intended to be common for for any NASA mission software IV&V project • For a given NASA mission, develop test plan using the System Reference Model (SRM) • For a given NASA mission flight software test article: • Execute it • Stimulate it according to IV&V test plan • Observe its behavior

7. Approaches to Executing SUT • Mimic the intended target environment • Acquire COTS emulators for the target processors • Develop simulators • Leverege simulators already created by the developer • Use the development test environment • Connect remotely • Install our test environment at the development test site • Don’t execute • Analyze the developer’s test results

8. Approaches to Stimulating SUT • Generate test scenarios from the system behavior models for a NASA mission found in theSRM • Scope the type of scenarios based on the results from validating the development test design • Install and use developers simulation models in our IV&V test environment • Create our own simulation models to stimulate the software • Don’t stimulate • Developer incorporates IV&V test suite into their test plan and IV&V analyzes the test results

9. Approaches to Observing SUT • Manually inspect the test results • Monitor the behavior of the SUT using formal mechanisms such as state chart assertions created by the SRM and validation teams • On-line during test • Off-line after test, based on test results

10. Testing Strategies • A strategy is a plan of action designed to achieve a particular goal1 • The expected goal of an analyst using the system: • To determine whether or not the implemented system behaves as we expect it to, doesn’t behave like we don’t what it to, and to ensure it behaves as it should under adverse conditions. • Possible strategies • Improvement to the development test design • Analyze the development test results • Supplement the development test with our test suite • Test independent of the developer 1: From the Oxford English Dictionary

11. Limitations exist in developer’s test design Limitations do not exist in developer’s test design Develop Software Test Suite Analyze Development Test Results Development test facility is available for IV&V Development test facility is not available for IV&V Integrate IV&V Test Environment with Development Facility Configure IV&V Test Environment Execute Software Test Suite at Development Test Facility Execute Software Test Suite at IV&V Facility Analyze IV&V Test Results

12. Test Environment Characteristics • Usable – common user interface across all IV&V projects • Adaptable – allow for the reconfiguration of testing resources for a individual IV&V projects • Extensible – allow for the future integration of new simulation tools, simulation models, and development test articles • Accessible • Interface with test monitors (e.g assertion state chart repositories) • Interface with development laboratories • Portable – for the possibility of testing at development site

13. Status • Vision and Concept of Operations documents are baselined • Took inventory of available hardware and software resources at the facility • Created first draft of cost estimate and acquiring vendor quotes • Working on a proof of concept to explore the process of dynamically testing flight software as a form of software implementation verification. • The prototype will be used to improve our Concept of Operations and validate our Vision. • Using uncompiled mission code that was already provided by developer

14. Challenges • Collecting all the information and materials needed to develop the test suite and configure the test environment in a timely manner. This may take multiple iterations • Configuring the test environment with the necessary simulators and emulators to successfully run and stimulate the software under test