Integrating Software into PRA Presented by C. Smidts Center for Reliability Engineering University of Maryland

1. Integrating Software into PRA Presented by C. Smidts Center for Reliability Engineering University of Maryland Center for Reliability Engineering

2. Integrating Software into PRA Probabilistic Risk Assessment (PRA) is a technique to assess the probability of failure or success of a mission. Current PRA neglects the contributions of software to the risk of the mission. The objective of our research is to extend current PRA methodology to integrate software in the risk assessment process. Center for Reliability Engineering

3. What We Have Done to Date • Built a Software Failure Mode Taxonomy • Failure Modes’ Quantification: generic high-level data • Public Literature • Expert Opinion • Collaborated with JSC through Ms. Alice Lee • Validate Our Methodology • Collect Data • Developed a Test-Based Methodology for Integrating Software Into PRA Center for Reliability Engineering

4. What We Are Planning to Do in the Future • Investigate Scalability Issues of the Test-based Approach • Continue the validation of our methodology with JSC • Apply the approach to JSC system • Revise the methodology based on NASA system • Develop an Analytical Approach • Apply the Analytical Approach to JSC system • Revise the Analytical Approach based on NASA system Center for Reliability Engineering

5. Integrating Software into PRA: A Test-based Approach Presented by C. Smidts C. Smidts, B. Li, M. Li Center for Reliability Engineering University of Maryland Center for Reliability Engineering

6. Integrating Software into PRA - Approach • We are working on an approach to integrate software into PRA. • Step 1: Identify events/components controlled/supported by software in MLD, accident scenarios, fault trees. • Step 2: Specify the functions involved • Step 3: Model software functions in ESDs/ETs and Fault Trees • Step 4: Construct the input tree • Step 5: Quantify the input tree • Step 6: Develop and perform software safety tests Center for Reliability Engineering

7. Example System An exit system in a building is used as the example in this case study. The exit system includes an emergency exit system and the PACS system. The Emergency exit system includes an emergency exit door and a marked egress router. It provides an escape route for personnel located inside the building during emergency situations. The PACS system is a simplified version of an automated personal entry/exit access system used to provide privileged physical access to rooms /buildings, etc. Personal ID and PIN are needed to access this system. Center for Reliability Engineering

8. Integrating software into PRA - Approach Step 1: Identify events/components • Identify events/components controlled/supported by software in MLD, accident scenarios, fault trees. • For all such events, create/expand contributors to account for software. • Verify that no neglected “events” may now have become possible due to software. Center for Reliability Engineering

9. MLD Center for Reliability Engineering

10. MLD Center for Reliability Engineering

11. Accident Description • Fire is the initiating event • Response systems: Emergency system and PACS system • End State: Loss of life Center for Reliability Engineering

12. Center for Reliability Engineering

13. Integrating software into PRA - Approach Step 2: Specify the functions involved • Not all software functions are involved in accident scenarios, i.e, not all software functions are involved in particular scenarios/fault trees or even in the entire realm of possible scenarios/fault trees. • To identify the specific functions involved in a scenario, determine the specific input to/output from the software – this will describe one function. • A list of possible functions can be found in the requirements. • Match the input/output combinations of these functions to the risk model Center for Reliability Engineering

14. Integrating software into PRA – Approach PACS Functional Decomposition Center for Reliability Engineering

15. Integrating software into PRA - Approach Actions and their inputs and outputs Center for Reliability Engineering

16. Integrating software into PRA - Approach Step 3: Modeling software function in ESDs/ETs and Fault trees • In the ESDs/ETs, the function of interest is modeled as Center for Reliability Engineering

17. Integrating software into PRA - Approach Step 3: Modeling software function in ESDs/ETs and Fault trees • In the fault tree, the function of interest is modeled as Center for Reliability Engineering

18. Integrating software into PRA - ESD Center for Reliability Engineering

19. Integrating software into PRA - Approach Step 4: Input Tree • Build the input tree for the particular function involved • The input tree is a decomposition of the space of possibilities • The input tree is mostly generic for a function. But may VARY due to context.(i.e. probabilities of basic events may vary, certain events may conflict with the rest of the scenario conditions.) Center for Reliability Engineering

20. Integrating software into PRA - Approach Step 4: Input Tree Center for Reliability Engineering

21. Input Fault Tree Input Fault Tree for SW1 Center for Reliability Engineering

22. Input Fault Tree Center for Reliability Engineering

23. Integrating software into PRA - Approach Step 5: Quantify the input tree Center for Reliability Engineering

24. Integrating software into PRA - Approach Step 6: Develop and perform software safety tests • These tests’ unique objective is to answer the questions contained in the model, i.e. in the MLD, accident scenarios and fault tree. • The test is completely automated using Test Generation/test execution tools (TestMaster/WinRunner). • The process is as follows: • Build a finite State Machine model of the software by following the software functional decomposition derived from the risk model and the software requirements. • Derive the test profile and output conditions to be quantified from the risk model • Define and run the test cases according to the following test strategy • Analysis consists in computing the probabilities of the different outcomes based on the test data. Center for Reliability Engineering

25. TestMaster Model Center for Reliability Engineering

26. Test Script Example win_activate ("mmount-76.umd.edu - CRT"); start_time1= get_time(); type ("1<kReturn>"); Check_Message(Message_b,1); type ("0<kReturn>"); Check_Message(Message_c,1); type ("155721495<kReturn>"); Check_Message(Message_b,1); type ("0<kReturn>"); Check_Message(Message_c,1); type("GayyardLupieN<kReturn>"); Check_Message(Message_b,1); type("1<kReturn>"); end_time1=get_time(); report_msg("Cardtime is "&(end_time1-start_time1)"Seconds"); start_time2= get_time(); Check_Message(Message_d,1); wait( 9); type("4"); Check_Message(Message_e, 1); wait(3); type("5"); Check_Message(Message_f, 1); wait(1); type("1"); Check_Message(Message_g, 1); wait(3); type("9"); end_time2=get_time(); report_msg("PINtime is "&(end_time2-start_time2)"Seconds"); Case_Judge(Message_a,1); Center for Reliability Engineering

27. Test Profile Test Profile for PACS Center for Reliability Engineering

28. Failure modes application • Test Case Selection • Sample from the profile/input tree to see whether we have a “Normal” or an “Abnormal Input”. • If it is a normal input, select randomly from the “Normal Input” domain. • If it is an abnormal input, randomly select the failure mode according to the profile/input tree. • Then randomly select the “base”value from the “Normal Input” domain and mutate this “base” value using the rules given below: Center for Reliability Engineering

29. Testing Results 200 cases have been tested for SW1 and SW2. 19 cases failed. Failed cases classification SW1 fails in only one case (58). Therefore, the point-estimate probability of Card failure is 1/200=0.005. 18 cases failed for SW2. Therefore, the unsafe probability (gate closed) is 18/199 =0.09. Center for Reliability Engineering

30. Testing Results Card time and Probability Center for Reliability Engineering

31. Testing Results PIN time and Probability Center for Reliability Engineering

32. Test Cases Coverage Input Failure Modes Coverage (SW1) Input Failure Modes Coverage (SW2) Center for Reliability Engineering

33. ESD Center for Reliability Engineering

34. Future Work • Represent hardware-related input failure modes in test model • Quantification of input fault tree based on field data • Output failure modes/Support failure modes • Sensitivity Analysis • Scalability • Test case generation • Test case execution • Number of test cases for each software component Center for Reliability Engineering