Injecting Faults for Error Evaluation

1. Injecting Faults for Error Evaluation NASA Glenn Research Center Kalynnda Berens, SAIC Richard Plastow, SAIC

2. Applications often consists of software components plus custom development, merged into a coherent package. COTS, GOTS, open source, etc. Source code is usually not available for review of quality and reliability. Visibility into the component is only what’s available via a public interface What is the quality of that component? What faults lay inside the component? Applications interface with hardware and other software and can be influenced by failures in those systems. Introduction SAS 2004 - Fault Injection

3. Fault Injection on Interfaces • Interfaces (hardware, software, human) are a major source of errors and induced faults • Software and system testing looks at anticipated off-nominal situations, but often misses unusual situations or combinations of faults • Mishap investigation has shown that multiple faults or unexpected anomalies are key players in accidents and mission failures SAS 2004 - Fault Injection

4. Example System System Hardware COTS Library Application Other Applications on same system Input Sensors Control Outputs COTS Operating System External Systems SAS 2004 - Fault Injection

5. Obtain Source Code and Documentation Identify Interfaces and Critical Sections Start Error/Fault Research Sufficient time and funds? Estimate Effort Required Select Subset Importance Analysis Yes Feedback to FCF Project Fault Injection Testing Test Case Generation End Document Results, Metrics, Lessons Learned Fault Injection Flow Diagram No Results SAS 2004 - Fault Injection

6. Interface Identification • Artifacts and Documentation • Software and System Requirements and Design specifications • Interface Specifications • User and Training Manuals • Hardware Documentation • Other project documentation • For FCF, “Signals List” • Source code SAS 2004 - Fault Injection

7. Error Research • Sources of Error/Fault Information • Vendor documentation • Public bug list • Internet Sources • Software logs • Error databases • Project Experience • Previous Test Results • Personnel Experience SAS 2004 - Fault Injection

8. Estimation of Effort • Determine level of effort, funding, time constraints • If complete effort not possible • Perform importance analysis of interfaces, software units • Safety • Complexity • Use by other system elements • Expected number or types of faults • Prioritize and select by importance SAS 2004 - Fault Injection

9. Testing • Test case generation based on identified errors plus permutations on possible input values • Consider multiple faults • Consider faults while system is off-nominal from a previous fault • Consider effects of system load/stress • Consider state-specific effects • Instrument software to observe effects of injected faults • External or observable effects • State changes (or lack of) • Effects on safety-critical functions SAS 2004 - Fault Injection

10. Results: First Project: Tempest • Written in Java 1.1 • Configurable • Cross platform operability • Implements HTTP GET and HEAD Request and Server Side Includes • Has some Basic Security Features • Debug Mode monitoring • Commercially available SAS 2004 - Fault Injection

11. Tempest Critical Errors • Inappropriate system operation with modified configuration file • Non-compliance with HTTP standard • System crash with invalid port numbers • Port 49151.45 -> opened port 80 • File access in server machine outside of authorized directories • System did not operate as per user documentation SAS 2004 - Fault Injection

12. Results: Fluids and Combustion Facility • Permanent, multi-user facility for ISS microgravity experiments • Fluids Integrated Rack (FIR) and Combustion Integrated Rack (CIR) • Operates for 10 years, so robustness important • CANbus processors selected for fault injection • Health and Status Monitoring • Cannot be upgraded in flight • Mature requirements, design, and interface definition • Source code available SAS 2004 - Fault Injection

13. CANbus Processors • Air Thermal Control Unit (ATCU) • Color Camera Package (CCP) • FOMA Control Unit (FCU) • FSAP Diagnostic Board • Input/Output Processor (IOP) • IPSU Diagnostic Board* • Mass Data Storage Unit (MDSU)* • Nd:YAG Laser Package* • Water Thermal Control System (WTCS) • White Light Package * Not yet tested SAS 2004 - Fault Injection

14. ECS CANbus Input-Output Processor (IOP) IOP Main Processor IOP HRDL Processor FSAP IOP CAN Node Processor FSAP Main Processor IOP Video Switch Processor FSAP CAN Node Processor Ethernet Optics Bench CANbus MDSU Common IPSU IPSU Main Processor MDSU Main Processor ISPU CAN Node Processor MDSU CAN Node Processor FIR System Diagram PI Package ATCU CAN Processor WTCS CAN Processor Laser Diode CAN Processor Nd:Yag CAN Processor White Light CAN Processor DCM CAN Processor SAS 2004 - Fault Injection

15. Initialization Off-Nominal (O-N) Operational (OP) Power Down (P) CANbus Processor State Diagram Power On Error Operational Cmd Success Error Power Down Cmd Error Operational Cmd Power Down Cmd Power Off SAS 2004 - Fault Injection

16. Testing Software SAS 2004 - Fault Injection

17. Test Setup SAS 2004 - Fault Injection

18. FCF Fault Injection Process • Interface Identification and prioritization • Obtain hardware, source code for testing environment • Error/Fault search on selected interfaces and components • Static analysis using Understand™ tool • Analysis of previous testing, defects • Test case generation, source code instrumentation, and test execution SAS 2004 - Fault Injection

19. Types of faults injected • Out-of-range • Unexpected input • Multiple errors • Timing • Flood the input with values • Remove Input/Output • Interrupt Input/Output SAS 2004 - Fault Injection

20. FCF Results • Software previously qualified • 35 errors, 3 critical • Loss of the output connection caused a continuous reboot • Changing the processor address caused a hang condition • Going to the input limits caused invalid telemetry to be sent. • Project corrected 20 errors • 4 errors still in process • Testing still in progress SAS 2004 - Fault Injection

21. Final Steps • In-depth case study (ISS flight payload) • Update Fault Injection Methodology document • Record all the details – problems as well as successes • Compare results to other defect detection mechanisms • Written for those who want to try the technique • Release FI Methodology and Case Study • December, 2004 SAS 2004 - Fault Injection

22. Passing the torch • Potential applications • Any software project using COTS software or with hardware interfaces • Data and Case Studies • Fault Injection Methodology (draft) • Available through SARP • Case Study (FCU main processor) • Available December, 2004 SAS 2004 - Fault Injection