FY 2004 Initiative: Risk Assessment of Software Architectures

1. FY 2004 Initiative: Risk Assessment of Software Architectures Less risk, sooner WVU UI: Risk Assessment of Software Architectures Hany Ammar, Katerina Goseva-Popstojanova,Ajith Guedem, Kalaivani Appukkutty, Walid AbdelMoez, and Ahmad Hassan LANE Department of Computer Science and Electrical EngineeringWest Virginia University

2. Outline • Introduction • Severity Analysis Methodology • Requirement Risk Analysis Methodology • Relevance to NASA Case studies • Conclusion

3. Introduction • Risk is a combination of: Theprobability of failure and the consequence of the failure (Severity) • Severity analysis is conducted on Unified Modeling Language (UML) models to quantify the severity of failure • Severity is estimated based on cost of failure and Hazard analysis techniques: • Functional Failure Analysis (FFA) • Failure Mode and Effects Analysis (FMEA) • Fault Tree Analysis (FTA) • Requirement Risk is assessedbased on: Probability of failure (Normalized Dynamic Complexity) andSeverity

4. Outline • Introduction • Severity Analysis Methodology • Requirement Risk Analysis Methodology • Relevance to NASA Case studies • Conclusion

5. Proposed Severity Analysis Methodology • INPUT: DYNAMIC UML SPECIFICATIONS • System level scenarios related to use cases, annotated with the following: • Guide words (Omission, Commission, Late, etc) • Cost of hazard • Sequence diagram that show the component/connector interactions annotated with the following: • Component states, probability that a component fails in a given failure mode, and the cost of failure in each state • Messages transmitted between components, probability that connectors fail to transmit a message, and cost of failure of each message • OUTPUT: COMPONENTS/CONNECTORS and SCENARIO SEVERITY

6. 1 Proposed severity analysis methodology 4 FFA Use Case Diagrams, System level Scenario Diagrams Scenario level Cost-Severity Graph (List of scenario level Hazards) Scenario Severity And Components/ Connectors Severity 3 Cost of Failure Graph 5 UML Specs FTA Component/ connector Cost-Severity Graph Sequence Diagrams Component/ Connector interactions (List of Component/Connector Failure Modes) FMEA 2 Scenario Severity Component/Connector Severity

7. Proposed Severity Analysis Methodology • Identify system hazards: states of the system that can contribute to accidents and mishaps Perform FFA using system scenario diagrams as an input • Identify component/connector failure modes Perform FMEA using UML Sequence Diagrams as an input • Construct a detailed cause-and-effect model, to record how failures propagate from component/connector level to the system level FTA is used to combine the outputs from FFA and FMEA • Develop the Cost of Failure Graphto estimate severity of each component/connector in a given scenario, or severity level of the scenario • Estimate severity of component/connector and system scenario using cost of failure graph and cost-severity graph The final result is a table of component/connector severity and system scenario severity.

8. Step 1: FFA Table

9. Step 2: Example (Pacemaker AVI Scenario) Sequence diagram of the AVI scenario

10. Step 2: FMEA Table

11. Step 3: Fault Tree Analysis (FTA) Step3: Combining the results of steps 1 and 2 to build a cause-effect model by applying FTA Step 1:Top event hazard identified by applying FFA Step 2: Component/Connector failure modes identified through FMEA Commission “Pace” Fault Tree

12. Step 4: Cost of Failure Graph forAR Component System Level hazards (Fault Trees Top Event) Failure Modes probabilities of AR Component Consequence (Cost) of AR Failure Modes “Command” Value Error $ 1000 (Regular Care) AR failed to handle ToOn P(“failed to handle ToOn”) =.09 P(“Command” Value Error)=0.02 $ 1000 (Regular Care) AR “stuck in Refractory” State P(“stuck in Refractory”) = 0.05 Commission “Pace” AR Cost Of Failure P(Commission “Pace”) = 0.65 Sense TimeOut Error P(“Sense TimeOut Error”) = 0.1 $ 100000 (Intensive Care) AR “ stuck in Waiting” State P(“stuck in Waiting”) = 0.05 $ 100000 (Intensive Care) Omission “VSense” Sense TimeOut Error P(“Sense TimeOut” Error) = 0.1 P(Omission “VSense") = 0.5 $ 100000 (Intensive Care) AR “stuck in Pace” State P(“stuck in Pace”) = 0.03 $ 100000 (Intensive Care) PaceTimeOut Error P(“PaceTimeOut Error”) = 0.03

13. Step 5: Severity of components/connectors Cost-Severity Graph

14. Step 5: Component/connector severity and scenario severity AVI scenario severity Severity of components/connectors in the AVI scenario

15. Outline • Introduction • Severity Analysis Methodology • Requirement Risk Analysis Methodology • Relevance to NASA Case studies • Conclusion

16. Impact Matrix (DDP Process) Proposed Requirement Risk Matrix Risk factor of scenario S1 in Failure mode FM2 • Requirements are mapped to UML use case and scenarios • A failure mode refers to the way in which a scenario fails to achieve its requirement • Failure mode of a scenario is determined based on the severity methodology • According to Dr. Martin Feather’s DDP Process, “The Requirements matrix maps the impacts of each failure mode (should it occur) on each requirement.”

17. Calculating Risk Factor • The dynamic complexity is calculated at the scenario level • The risk factor of a particular scenario (a), in a particular failure mode (b) is: Rfab = Normalized Dynamic Complexityab X Severityab • Severity of a scenario in a specific failure mode is estimated using the previously discussed severity analysis methodology [Slide 6]

18. Calculating Complexity • We estimate complexity as: Complexity = CC * Msg • CC : McCabe’s cyclomatic complexity of a scenario for each failure mode • Msg: The number of messages exchanged between the components in the sequence diagram to the point where the failure mode occurs in the scenario According to Fenton et al, “The product of cyclomatic complexity (CC) and sigFF was shown to be a good predictor of fault proneness. ..The combined metrics appear to be better than both sigFF and Cyclomatic complexity on their own and also better than the size metric”

19. Proposed Requirement Risk Analysis Methodology • STEP 1:Map the requirements to UML sequence diagrams • For each Sequence diagram • STEP 2:Construct the control flow graph of the scenario from the sequence diagram • STEP 3:Identify the different failure modes on the sequence diagram and the control flow graph • For each failure mode • STEP 4:Assess the severity of the failure mode using Function Failure Analysis (FFA) • STEP 5:Determine the cyclomatic complexity of the sub-control flow graph • STEP 6: For the failure mode, measure the number of messages exchanged between the components in the sequence diagram. • STEP 7:Calculate the complexity of the scenario for the failure mode as: • Cyclomatic complexity * Number of messages • STEP 8:Calculate the Risk factor of the scenario for the failure mode as: Complexity * Severity • End For each failure mode • End for each sequence diagram

20. Requirement R2 of Pacemaker : Mapped to Atrial-Ventricular Inhibit Scenario FM1 Msg = 4 FM2 Msg = 7 FM3 Msg = 9 Atrial-Ventricular Inhibit Scenario: Control Flow Graph (Obtained from Sequence Diagram) Failure Modes and Msg Values marked in the sequence diagram

21. Calculating Complexity Severity index of failure modes FM1, FM2 and FM3 of AVI scenario is 0.95 Risk is calculated as the product of Complexity and Severity index Calculating Risk

22. Requirement Risk Matrix for Pace Maker

23. Outline • Introduction • Severity Analysis Methodology • Requirement Risk Analysis Methodology • Relevance to NASA Case studies • Conclusion

24. Severity Analysis of EOS • Severity Analysis Methodology is applied on NASA’s Earth Observing System (EOS) Preplanned emergency scenario

25. FFA Table for EOS

26. Requirement Risk Matrix for HCS case study

27. Outline • Introduction • Severity Analysis Methodology • Requirement Risk Analysis Methodology • Relevance to NASA Case studies • Conclusion

28. Conclusion • We have developed a methodology to assess severity of failures of components, connectors, and scenarios based on UML models • We have developed a methodology for assessing requirements based risk using normalized dynamic complexity and severity of failures • These methodologies were applied on NASA case studies • Our future work will address the validation of risk methodology using software artifacts and data from the Metrics Data Program