Towards a Radically New Theory of Software Reliability

1. Towards a Radically New Theory of Software Reliability Aditya P. Mathur Head and Professor Department of Computer Science, Purdue University ABB, Sweden Monday April 7, 2008

2. Reliability Probability of failure free operation in a given environment over a given time. Mean Time To Failure (MTTF) Mean Time To Disruption (MTTD) Mean Time To Restore (MTTR)

3. Claim Existing theories of software reliability simplify the problem to the extent that they (almost) maximize the uncertainty associated with the estimated software reliability.

4. Operational profile Probability distribution of usage of features and/or scenarios. Captures the usage pattern with respect to a class of customers.

5. Decision process Random or semi-random Test generation Reliability estimation [Uncertainty evaluation?] Test execution Failure/Defect data collection Reliability estimation Operational profile

6. Issues: Operational profile Variable. Becomes known only after customers have access to the product. Is a stochastic process…a moving target! Random test generation requires an oracle. Hence is generally limited to specific outcomes, e.g. crash, hang. What about an operational profile with impulse? This creates a non-differentiable probability function of the time-to-failure.

7. Issues: Failure data Should we analyze the failures? If yes then after the cause is removed then the reliability estimate is invalid. If the cause is not removed because the failure is a “minor incident” then the reliability estimate corresponds to irrelevant incidents.

8. Issues: Failure rate “That is, the failure rate, when unambiguously defined, does not have a physical reality; rather, it is a technical device, whose sole purpose is to convey the engineer’s personal opinion about the life characteristic of software.” Nozer Singpurwalla, “The failure rate of software: does it exists?”, IEEE Transactions on Reliability, vol. 44, no. 3,1995.

9. Issues: Model selection Rarely does a model fit the failure data. Model selection becomes a problem. 200 models to choose from? New ones keep arriving! Markov chain models suffer from a lack of estimate of transition probabilities. • To compute these probabilities, you need to execute the application. • During execution you obtain failure data. Then why proceed further with the model?

10. C1 C3 C2 Issues: Markovian models 12 12 + 13=1 21 32 13 Markov chain models suffer from a lack of estimate of transition probabilities. • To compute these probabilities, you need to execute the application. • During execution you obtain failure data. Then why proceed further with the model?

11. Issues: Assumptions Software does not degrade over time; memory leak is not degradation and is not a random process; a new version is a different piece of software. Reliability estimate varies with operational profile. Different customers see different reliability. • Can we not have a reliability estimate that is independent of operational profile? • Can we not advertise quality based on metric that are a true representation of reliability..not with respect to a subset of features but over the entire set of features?

12. Estimating Uncertainty Estimates of software reliability must the associated with uncertainty. But how to quantify uncertainty? • Entropy based approach [Katerina et al. 2002] • Moments based approach [Katerina et al. 2003] • Monte Carlo approach [Katerina et al. 2003] • Bayesian approach [Dai et al. 2007]

13. Estimating Uncertainty Basic idea: • Model the parameters as random variables. • Use statistical (e.g. moments) or Simulation approaches to estimate variance. Problem: • Does not correlate with likely faulty components in the program under test.

14. Risky Desirable high Reliability low Undesirable Suspect model low high Coverage Problem with existing approaches to reliability estimation. Sensitivity of Reliability to test adequacy

15. Basis for an alternate approach Why not develop a theory based on coverage of testable items and test adequacy? Testable items: Variables, statements,conditions, loops, data flows, methods, classes, etc. Pros:Errors hide in testable items. Cons:Coverage of testable items is inadequate. Is it a good predictor of reliability? Yes, but only when used carefully. Let us see what happens when coverage is not used or not used carefully.

16. u4 u2 u1 u3 True reliability (R) Estimated reliability (R’) Saturation region Saturation Effect R’m R’d R’df R’f Reliability Rm Rdf Mutation Rd Dataflow Rf Decision Functional tfs tfe tds tde tdfs tdfe tms tfe Testing Effort u:uncertainty FUNCTIONAL, DECISION, DATAFLOW AND MUTATION TESTING PROVIDE TEST ADEQUACY CRITERIA.

17. An experiment [TeX] Tests generated randomly exercise less code than those generated using a mix of black box and white box techniques. Application: TeX. Creator: Donald Knuth. [Leath ‘92]

18. An experiment [sort utility] UNIX sort utility [DelFrate et al. 1995]

19. An experiment [coverage-reliability correlations] Unix utilities and space application [Garg 1995. MS Thesis]

20. Component Component Component Component Interactions Interactions ………. Component Component OS Component Component Component Interactions Modeling an application

21. Reliability of a component Reliability, probability of correct operation, of function f based on a given finite set of testable items. R(f)= (covered/total), 0<<1. Issue: How to compute  ? Approach: High correlation between coverage metrics and failures has been established via empirical studies. Such studies could provide estimate of  and its variance for different sets of testable items.

22. Reliability of a subsystem C={f1, f2,..fn} is a collection of components that collaborate with each other to provide services. R(C)= g(R(f1), R(f2), ..R(fn), R(I)) Issue 1: How to compute R(I), reliability of component interactions? Issue 2: What is g ? Issue 3: Theory of systems reliability creates problems when (a) components are in a loop and (b) are dependent on each other.

23. Scalability Is the component based approach scalable? Powerful coverage measures lead to better reliability estimates whereas measurement of coverage becomes increasingly difficult as more powerful criteria are used. Solution: Use component based, incremental, approach. Estimate reliability bottom-up. No need to measure coverage of components whose reliability is known.

24. Next steps Develop component based theory of reliability. Base the new theory on existing work in software testing and reliability. Do experimentation with large systems to investigate the applicability of the their and its effectiveness in predicting and estimating various reliability metrics.

25. Apple Confidence: 0.999 Level 0: 1.0 Level 1: 0.9999 Level 2: 0.98 The Future Boxed and embedded software with independently variable Levels of Confidence. Mackie Confidence: 0.99 Level 0: 1.0 Level 1: 0.9999

26. Select References M. H. Chen. A. P. Mathur, and V. J. Rego. A Case Study To Investigate Sensitivity Of Reliability Estimates To Errors In The Operational Profile, Proceedings of the Fifth International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Monterey, California, November 6-9, 1994, pp 276-281. P. Garg. On code coverage and software reliability. MS Thesis. Department of Computer Science, Purdue University. May 1995. F. Del Frate, P. Garg, A. P. Mathur, and A. Pasquini. On the Correlation Between Code Coverage and Software Reliability, Proceedings of the Sixth International Symposium on Software Reliability Engineering, IEEE Press,Toulouse, France, pp 124-132, October 24-27, 1995 S. Krishnamurthy and A. P. Mathur. On the Estimation of Reliability of a Software System Using Reliabilities of its Components, Proceedings of the 8th International Symposium on Software Reliability Estimation, Albuquerque, New Mexico, November 1997. Katerina Goseva–Popstojanova and Sunil Kamavaram. Assessing Uncertainty in Reliability of Component–Based Software. Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03), 2003. Yuan-Shun Dai and Min Xie and Quan Long and Szu-Hui Ng. Uncertainty Analysis in Software Reliability Modeling by Bayesian Analysis with Maximum-Entropy Principle, IEEE Trans. Softw. Eng.,V 33, No. 11, 2007, pp 781--795.