Product-Line Verification of Safety-Critical Software

1. National Aeronautics and Space Administration Elliott, JPL: MSAP NASA, GRAIL Product-Line Verification of Safety-Critical Software Technical Presentation NASA OSMA Software Assurance Symposium September 8-12, 2008 Robyn Lutz, JPL & ISU This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, and at NASA Ames Research Center, under a contract with the National Aeronautics and Space Administration. The work was sponsored by the NASA Office of Safety and Mission Assurance under the Software Assurance Research Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance and Technology Program Office SAS08_Product_Line_Verif_Safety_Lutz

2. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical SoftwareProblem • Industrial experience shows that it is the verifiable conformance of each system to the product-line specifications that makes or breaks the product-line practice • Verification that the software for each project satisfies its intended product-line constraints is thus essential • How should we verify that delivered software conforms to the product-line and how do we document that conformance? • How should we verify that safety-critical software built using product-line assets is safe? SAS08_Product_Line_Verif_Safety_Lutz

3. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Approach • Identify candidate suite of product-line verification techniques, focusing on safety • Tailor as appropriate to NASA needs • Apply, evaluate, and report results in collaboration with project • Produce examples and data for training and transfer of techniques to projects SAS08_Product_Line_Verif_Safety_Lutz

4. Product-Line Verification of Safety-Critical SoftwareDefinition of product line National Aeronautics and Space Administration A software product line is defined to be “a set of software-intensive systems sharing a common, managed set of features that satisfy the particular needs of a specific market segment or mission and that are developed from a common set of core assets in a prescribed way” [Clements and Northrop, 2002]. SAS08_Product_Line_Verif_Safety_Lutz

5. Product-Line Verification of Safety-Critical SoftwareOverview National Aeronautics and Space Administration • MSAP Project: Worked this year with Multi-Mission System Architecture Platform project at JPL • to identify (1) “negative lessons learned” from industrial use of product lines (risks & challenges) and (2) relevant mitigation strategies • findings requested by MSAP managers to use in product-line business case • MSAP work slowed when key personnel moved off project • GRAIL Project: Now working with Gravity Recovery and Interior Lab (Discovery Lunar mission, 2011; twin spacecraft; inheritance from GRACE’s earth-study techniques and from Lockheed Martin’s product-line spacecraft, especially MRO) • to perform trend-tracking for GRAIL of software changes and anomalies across product line (beginning with MRO): on-going in FY09 • to evaluate use of DDP to customize product-line risks/mitigations to GRAIL: on-going in FY09; collaborative with M. Feather SAS08_Product_Line_Verif_Safety_Lutz

6. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical SoftwarePlanned Application on GRAIL • GRAIL spacecraft is the latest in a LMA product line: high degree of software inheritance with some key differences (e.g., platform, lunar mission) • Industrial product-line verification techniques help predict quality/defects in a new system in a PL (such as GRAIL) based on similarities/differences with earlier systems in that PL (MRO, Phoenix, etc.) • Application to GRAIL: what can PL approach to GRAIL tell us about what to expect in terms of quality/defects/concerns for software verification? SAS08_Product_Line_Verif_Safety_Lutz

7. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Quality of data allows generalization of recommendations • Report : “Survey of Product-Line Verification and Validation Techniques” (2007) • Best practices extracted from reports of industrial experience • Verification challenges & verification enablers identified for NASA product lines based on experts’ recommendations • List of resources assembled (conferences, industrial and defense industry experiences, annotated bibliography) • Report: “Tool-Support Survey for Product-Line Verification and Validation Techniques” (2007) • Commercial and academic tools for product-line development • Configuration management and change management tools for product lines • Tool-supported testing in product lines (both domain testing and application testing) • Advice from product-line experts regarding tool support SAS08_Product_Line_Verif_Safety_Lutz

8. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Accomplishments this year • Report : “An Evaluation Report for Three Product-Line Tools ”, Simon Chong (6/08) • Widespread agreement that better tool support for product lines is needed for project applications • Report evaluated the functionality, usability and performance of 3 candidate product-line tools (2 commercial, 1 academic) identified in Task 2 • FORM (from Pohang University of Science & Technology), Pure::Variants (from PureSystems, GmbH), and Gears (from BigLever) • Downloaded and experimented with each tool • Emphasis on practical use: e.g., setup, support, user interface, support for commercial tools, documentation • Recommendation for application plan based on measured criteria: Gears SAS08_Product_Line_Verif_Safety_Lutz

9. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical SoftwareAccomplishments this year • Paper : “Enabling Verifiable Conformance for Product Lines” , R. Lutz • To be presented at 12th Int’l Software Product Line Conference, Sept. 8-12, 2008 • Recommendations for process characteristics that will make it easier to show that the system being built conforms to the product-line specifications • Draws on information from four product lines or product-line-like sets of systems (3 JPL, 1 NASA) • a set of interferometer (spaceborne telescope) software systems with reused architecture, documentation, and code • a proposed NASA mission consisting of a cluster of microsatellites • a multi-mission hardware and software architecture • a ground data system providing services for multiple spacecraft missions SAS08_Product_Line_Verif_Safety_Lutz

10. Product-Line Verification of Safety-Critical Software Accomplishments this year National Aeronautics and Space Administration • Paper describes three challenges for verifying conformance in NASA product lines: • Many will be safety or mission critical • Many will be long-lived • Many will be highly autonomous • Recommends process characteristics to handle challenges in the appropriate development phases • To establish verifiable conformance of safety requirements in the new product need rigorous traceability to the product-line safety requirements • To maintain conformance for long-lived missions need to accurately capture the requirements delta information • To verify conformance of highly autonomous product lines need good documentation of the product-line behavior, preferably as a machine-readable model , and use small configuration units SAS08_Product_Line_Verif_Safety_Lutz

11. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Synergistic activities • Collaborated Summer, 2007 with Jairus Hihn (JPL) & Dan Baker (WVU grad student withTim Menzies); Dan developed prototype of Maintenance Effort Estimation Tool • Served on program committees for Software Product Line Testing workshop, 2007, and Software Product Line Conference, 2008 SAS08_Product_Line_Verif_Safety_Lutz

12. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Findings: Enabling Verification of Inherited Software • Rigorous traceability • Key to being able to perform verification on product lines • Controlling requirements changes in baseline assets • Having this information in the application requirements specification allows scoping of features and code that will have to be developed beyond the product-line assets • Sufficient documentationis essential. • Verification that the new system conforms to the product-line requirements, architecture, and test suite requires good documentation of the new system. • Access to source code • For thorough testing of the product-line components, need source code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl] SAS08_Product_Line_Verif_Safety_Lutz

13. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Findings: Enabling PL Verification • Rigorous traceability • Key to being able to perform verification on product lines [Bergey]. • Capturing the requirements delta • Having this information in the application requirements specification allows scoping of features and code that will have to be developed beyond the product-line assets [Pohl] . • Sufficient documentationis essential. • Verification that the new system conforms to the product-line requirements, architecture, and test suite requires good documentation of the new system. • Access to source code • For thorough testing of the product-line components, need source code. Code and compilation and linking instructions need to be available in order to adequately verify the code [Pohl]. SAS08_Product_Line_Verif_Safety_Lutz

14. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical SoftwareFindings: Enabling PL Verification (cont.) • Start small • Successful product-line engineering was first applied on a legacy system “in which there is frequent change occurring at relatively high cost. Such a domain is often an isolatable section of the system where the changes can be encapsulated and where a single group of software developers is responsible for making the changes” [Weiss and Lai]. • Use small configuration units • Rockwell Collin described the tradeoff in their recent experience building a large product line: “Having more items leads to a finer grained decomposition of functionality, which, in turn, leads to higher levels of reuse and systems that are easier to test and get through airworthiness qualifications. However, having more items also means there are more configuration items to keep track of and maintain, more complexity, and a steeper integration curve” [Clements and Bergey]. SAS08_Product_Line_Verif_Safety_Lutz

15. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Findings: Tool Support for PL Verification • Build family-specific tools • “Invest in family-specific tools to make the production of family members rapid using the production process” [Weiss & Lai]. They recommend domain-specific modeling languages and code compilation or composition from the models. • Demand tool interoperability • Clements and Northrop emphasize that it is the integrated tool suite rather than the standalone tools that provide the needed development environment. “The interoperability of a chosen set of tools is critical for the automated production of products in a product line from core assets.” SAS08_Product_Line_Verif_Safety_Lutz

16. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical SoftwareFindings: Tool Support for PL Verification • Plan verification tasks to reuse test suites • Testing in a product line begins early as the domain-engineering product-line assets (the reusable components) are built. Then, each time those assets are assembled into a new product-line application, more testing occurs. “During application engineering, assets from this test infrastructure should be reused to speed up testing and thus overall product development” [Knauber and Hettrick]. • Make tools for configuration management (including variability management and change management) a high priority • The payoff for using a product line is safe and efficient reuse. To enable and control this reuse, product-line developers agree that tool-supported configuration management is required. The challenge is that, “Traditional CM tools are meant to handle variation over time, but they are not good at handling simultaneously supported variations.” [Clements and Northrop] SAS08_Product_Line_Verif_Safety_Lutz

17. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical SoftwareTechnical challenges of approach Challenges and techniques for verifiable conformance of a product line SAS08_Product_Line_Verif_Safety_Lutz

18. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Challenges for work to be more widely applicable • Organizational considerations • External ownership of product line development can make access to requirements documents, data, code, more difficult • Reluctance to invest for other projects • Projects that are willing to use other’s products but are reluctant to make investments for other projects can impede product-line development and verification. • Difficulty controlling the delta • CCB usually first line of defense in deciding whether new features, alternative design mechanisms, or different coding standards will move systems away from the product-line standard set of verified components. • Balancing commonality and common sense • Folding changes in one product back into the product line can lead to degradation of the product line and the erosion of commonality. On the other hand, propagating each change to the product line can make re-verification costs unmanageable. SAS08_Product_Line_Verif_Safety_Lutz

19. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Technology Readiness Level • TRL 3 • “Analytical and experimental critical function and/or characteristic proof of concept” • MSAP provided limited proof-of-concept of usefulness of product-line engineering verification techniques for inherited software • GRAIL application goal for next year: validation in relevant environment (TRL 5) SAS08_Product_Line_Verif_Safety_Lutz

20. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Relevance to NASA • NASA is reusing more product-line, inherited, & reused software assets • We need better ways to verify compliance of delivered software against the constraints levied on all systems in that product line • We need better ways to verify that problems with similar, earlier systems won’t recur in our new system • Work focuses on identifying product-line verification techniques that can help solve problems now for current missions SAS08_Product_Line_Verif_Safety_Lutz

21. National Aeronautics and Space Administration Product-Line Verification of Safety-Critical Software Planned capability of research Customize industrial, product-line techniques to verify correctness of a new system’s software inheritance, and apply to NASA systems: • GRAIL (FY09 focus) • JUNO (same heritage) • Future NASA spacecraft with same LMA product-line heritage • Exploration systems reusing same baseline and inherited software SAS08_Product_Line_Verif_Safety_Lutz