Sensitivity of Software Reliability to Operational Profile Errors

1. Real-World Software Reliability Assessment(WVU UI#7: Sensitivity of Software Reliability to Operational Profile Errors: Architecture-Based Approach) PI: Katerina Goseva – Popstojanova Students: Sunil Kamavaram & Olaolu Adekunle Lane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, WV katerina@csee.wvu.edu

2. What we are doing? Anyone can see a fire What we need are smoke detectors But what about the sensitivity and accuracy of the alarms ?

3. Problem statement & Our goal • Traditional view: Point estimate of software reliability computed from the model using point estimates of input parameters • Problem: Estimation of a trustworthy operational profile is difficult • IV&V information on operational profiles - limited, may be inaccurate • Single operational profile could not be sufficient to describe the use by different users • Software systems evolve - operational profile may change • Our goal: Reliability “sensitometer” that enables us to answer the question “How parameters uncertainty propagates into overall application reliability?” • Develop an architecture-based methodology for uncertainty analysis of software reliability & apply it on case studies

4. Reliability frequency chart & distribution fitting Entropy as a measure of uncertainty Execution rates & uncertainty of components Certainty bands (percentiles) What we can do? • Benefits to IV&V • Software reliability assessment throughout the life cycle (keeping track of the software evolution) • Allocation of testing efforts • Software certification

5. Uninformed Approach (maximum entropy) Intended Approach (historical data, UML) Informed Approach (component traces) Growth models Non-failed executions R1 1 p12 Fault injection R2 2 p23 1-p12 R3 1-p23 3 Uncertainty analysis 1 E Architecture - based methodology for uncertainty analysis

6. Probability distributions Uncertainty analysis Methods for uncertainty analysis Sensitivity studies Entropy Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis

7. Choice of the method • Choose the method using the following criteria • Data requirements & ability to collect data • Reliability measures • Accuracy of the solution • Scalability with respect to the number of components • Our goal: fill the table

8. 1 p12 2 p23 1-p12 1-p23 3 1 E Construction of the software architecture model • Structural phase – establishment of static software architecture • Software specifications • Architectural design • Parser-based or lexically based tools (SIAT tool - Titan Systems Corporation) • Statistical phase – estimation of the relative frequencies of component interactions, that is, transition probabilities • Uniform distribution – maximum entropy approach • Historical data • Software specification (e.g. UML use case & sequence diagrams) • Component traces from profiles or test coverage tools (Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU)

9. Informed Approach (component traces) R1 1 p12 Fault Injection (real faults) R2 2 p23 1-p12 R3 1-p23 3 1 E European Space Agency case study Component traces obtained during testing were used for constructing software architecture & estimating transition probabilities • Almost 10.000 lines of C code • The program has been extensively used after the last fault removal without failures; this gold version is used as an oracle Two faulty versions were obtained reinserting the real faults discovered during the integration testing and operational usage

10. Version p12 p23 A 0.5933 0.7704 B 0.7364 0.6866 Parameter estimation • Two versions • Version A: faulty components 1&2, fault-free component 3 • Version B: faulty components 2, fault-free components 1&3 • Transition probabilities where is the number of times control was transferred from component i to component j, and • Component reliability where is the number of failures and is the number of executions of component iin N randomly generated test cases

11. 1 1-R1 p12 R1 2 1-R2 F p23 R2 (1-p12)R1 3 (1-p23)R2 1-R3 R3 C 1 Construction of the architecture – based software reliability model E

12. Traditional View: Point estimates of software reliability • Actual reliability of the software where Fis the number of system failures in N randomly generated test cases • Estimated reliability from the model • Results

13. Probability distributions Uncertainty analysis Methods for uncertainty analysis Sensitivity studies Entropy Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis

14. Sensitivity of software reliability to variations in operational profile Version A reliability Version B reliability Rmax = 0.8414 Rmin = 0.7048 Rmax = 0.9983 Rmin = 0.8363

15. Probability distributions Uncertainty analysis Methods for uncertainty analysis Sensitivity studies Entropy Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis

16. Uncertainty study based on entropy • Entropy quantifies the uncertainty present in a stochastic source where represents the usage distribution and the transition probabilities • Higher entropy implies an exponentially greater number of statistically typical paths • Maximum entropy – all transitions that are exit arcs from each state are equiprobable

17. Uncertainty of the operational profile Hmax = 0.5514 Hmin = 0.0404 Hmax = 0.5514 Hmin = 0.0404 • Operational profile A (H=0.4707) is more uncertain than operational profile B (H=0.4604) • Software systems that have uniform operational profile are more uncertain and thus would require more testing

18. Uncertainty of software reliability Operational profile Version B uncertainty Version A uncertainty Version A reliability Version B reliability • Considering software failure behavior increases the uncertainty for both versions compared to the uncertainty due to operational profile • Version B, which is more reliable, is less uncertain than version A

19. Uncertainty of components for the operational profile • Uncertainty of component i is estimated using the conditional entropy • Uncertainty of component i will be higher if it transfers the control to more components and the transition probabilities are equiprobable Version A Version B

20. Uncertainty of components for the software reliability model • Uncertainty of component 1 version B remains the same because • For all other components uncertainty increases due to • Components that have higher expected execution rate, higher component uncertainty, and moderate reliability should be allocated more testing effort Version A Version B

21. Probability distributions Uncertainty analysis Methods for uncertainty analysis Sensitivity studies Entropy Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis

22. Uncertainty study based on the method of moments • Method of moments involves the following steps • Obtain the expression for the system reliability using the architecture-based software reliability model • Expand the expression for system reliability using Taylor series • Determine the moments of the components reliabilities • Estimate the mean and the variance of the system reliability using the parameter moments and Taylor series coefficients

23. First order Taylor series • First order Taylor series expansion where is the mean component reliability, and • Mean reliability is • Variance of the reliability is where is the variance of the component reliability

24. Second order Taylor series • Second order Taylor series expansion • Mean reliability is • Variance of the reliability is

25. Method of moments for the case study • Second order approximation does not improve accuracy significantly • Version B is more reliable with less variance of the reliability

26. Probability distributions Uncertainty analysis Methods for uncertainty analysis Sensitivity studies Entropy Analytical Monte Carlo simulation Method of moments Confidence intervals Perturbation analysis

27. Uncertainty study based on Monte Carlo simulation • Monte Carlo simulation involves the following steps • Obtain the expression for the system reliability using the architecture-based software reliability model • Assign probability distributions to the transition probabilities and components reliabilities • Sample the distributions • Compute the reliability of the system using the sampled values • Repeat steps 3&4 until the desired number of values of system reliability has been generated • Calculate the moments, frequency chart and percentiles for the system reliability, do the distribution fitting

28. Variation of the operational profile:Frequency chart and distribution fitting

29. 95% 75% Variation of the operational profile: Percentiles • 95% certainty band shows the range of values in which reliability has 95% chance of falling

30. Convergence of the mean Mean reliability =0.7600 The estimation of the mean reliability converges after around 3000 iterations

31. Variation of the operational profile: Sensitivity measured by contribution to variance • Reliability is more sensitive to p1E; the variance is positive • Reliability is also sensitive to p12; the variance is negative

32. Variation of the operational profile and component reliabilities: Frequency charts Version A Version B

33. Variation of the operational profile and component reliabilities: Distribution fitting & percentiles Version A Version B

34. Making a choice

35. Accomplishments • Architecture-based methodology for uncertainty analysis of software reliability was developed • Four different methods already developed • These methods were illustrated on the European Space Agency software

36. Future work • Develop other methods for uncertainty analysis • Complete “Make a choice” table • Apply & validate all methods using NASA case studies • SIAT tool - Titan Systems Corporation • Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU