Extension of E(Θ) metric for Evaluation of Reliability

1. Extension of E(Θ) metric for Evaluation of Reliability

2. Presentation • Background on Reliability and E(Θ) • Testing Methodologies • Present formulae for calculating E(Θ) • Need of a new formula • Formula Derivation • Validation of Derivation Technique • Summary and Future Work

3. E(Θ) ? • E(Θ) - Reliability of a software is defined in terms of its E(Θ) value. • E(Θ) refers to the probability of the software encountering a failure subsequent to testing

4. Why measure reliability • Cost of Designing a Faulty S/W in less time >> Cost of Designing a Reliable S/W • Safety Critical Applications

5. Faults or Bugs • Failure – Unexpected O/P • Faults – I/Ps causing Failures

6. How to measure reliability Testing: • Means of Probing S/W for bugs • Means of gaining confidence about its reliability Source : Evaluating Testing Methods by Delivered Reliability -Phyllis G. Frankl, Richard G. Hamlet, Member, IEEE, Bev Littlewood, Member, IEEE,and Lorenzo Strigini, Member, IEEE

7. Testing Methods • Operational – Test Case = Statistical Distribution or Random (I/P domain) • Debug – Test Case = Selective Sampling (I/P Domain)

8. Operational Vs. Debug • Debug – Probes for bugs actively (assumes S/W is faulty) • Operational – Waits for Failures to occur.

9. Sub-Domain Testing • A debug testing technique. • Concentrates test-cases to specific sub-domains with high failure probability • Based on Debugger’s Intuition • Eg. Boundary Testing – Bugs lie in the corners

10. Example Input domain + ve y + ve x 0 - ve x - ve y Failure Region Failure Region

11. Sub-Domain Testing So Far…

12. Terminologies used • q = failure probability • d = detection rate • T = Number of test cases per sub-domain • n = n number of sub-domains/failure regions

13. Single Domain - Single Failure Region Domain nature: Single Domain = 1 – 50 (50 points) Failure Region A = 21 – 40 (20 points) B, 30 A, 20 Failure Region E(Θ) = q( 1 - d)T

14. Single Failure Region – Multiple Sub-Domains Domain nature: Sub-Domain 1= 1 – 25 (25 pt.s) Sub-Domain 2= 25 – 50 (50 pt.s) Sub-Domain 3= 50 – 85 (35 pt.s) Failure Region A = 40-70 (30 pt.s) B, 20 C, 10 D, 25 Ai,5 Aj, 15Ak,10 E(Θ) = q nΠi=1 ( 1 - di)T

15. Multiple Failure Regions – Single Sub-Domain Domain nature: Domain = 1 – 100 (100 pt.s) Failure Region A = 20-30 (20 pt.s) Failure Region B = 50-70 (20 pt.s) Failure Region C = 80- 95(20 pt.s) D, 55 A, 10 B, 20 C, 15 E(Θ) = nΣi=1 qi( 1 - di)T

16. Multiple Failure Regions – Multiple Sub-Domains D, 30 Domain nature: Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 30-50 (10 pt.s) Failure Region A = 70-90 (20 pt.s) C, 30 B, 20 A, 20 E(Θ) = ? No definitive Formula. Empirical Testing only Solution

17. And Now……

18. New Set of Terminologies • q = Total failure probability • dA = Detection rate of a failure region in a sub-domain • Fi = Number of points in a failure region • M = Total number of failure points in the sub-domain

19. Multiple Failure Regions – Multiple Sub-Domains D, 30 Domain nature: Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 20-40 (20 pt.s) Failure Region A = 70-90 (20 pt.s) C, 30 B, 20 A, 20 E(Θ) = q – ( dAF1/M+ dBF2/M)

20. Formula Derivation D, 30 C, 30 Fault Case Regions Detected Faults Remaining 00 None A, B B, 20 A, 20 01 A B 10 A B 11 A, B None E (Θ) = (Probability that the particular case will occur) * (probability that the rest of the faults are undetected)

21. Contd. Assuming 1 test-case per sub-domain Fault A Test Case 2 Fault B Case Number Test Case 1 Region C undetected undetected Region D Seq. 1 detected Region D Region B undetected Seq. 2 undetected Region A Region C detected Seq. 3 detected Region A Region B detected Seq. 4 E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) contd.

22. E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} = {(30/50 * 30/50) * 40/100} + {(30/50 * 20/50) * 20/100} + {(20/50 * 30/50) * 20/100} + {(20/50 * 20/50) * 0/100} = 0.144 + 0.048 + 0.048 = 0.24

23. Verification of the Technique Domain nature: Domain = 1 – 100 (100 pt.s) Failure Region A = 20-40 (20 pt.s) Failure Region A = 70-90 (20 pt.s) C, 60 A, 20 B, 20 Approach I: E(Θ) = nΣi=1 qi( 1 - di)T = 20/100(1- 20/100) + 20/100(1- 20/100) = 0.32 Approach II: E(Θ) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} contd.

24. = {(80/100 * 80/100)(40/100)} + {(80/100 * 20/100)(20/100)}+ {(20/100 * 80/100)(20/100)} + {(20/100 * 20/100)(0/100)} = 0.32 Both approaches yield the same result Similar tests done for different scenarios validate the 2nd technique of calculating E(Θ) with respect to Frankl’s Method

25. Getting back to Formula Derivation Upon further solving the method just described we get ….

26. Method Reduction E (Θ) ={ (1- dA)* (1- dB)* (q) } + { (1-dA)* (dB)* (q –F1/M) } + { (dA)*(1- dB)* (q –F2/M) } + { (dA)* (dB)* (q –F1/M – F2/M) } = q [ {1–dB – dA + dAdB} + { (dB – dAdB) (1–F2/M*q) } + { (dA – dAdB) (1-F1/M*q) } + { dAdB (1–F1/M*q – F2/M*q) } ] Contd.

27. = q [ {1–dB –dA + dAdB} + { dB – dB*F2/M*q – dAdB + dAdBF2/M*q}+ { dA – dAF1/M*q – dAdB + dAdBF1/M*q} + { dAdB–dAdB*F1/M*q – dAdBF2/M*q} ] = q [ 1– dBF2/M*q – dAF1/M*q] = q – ( dAF1/M + dBF2/M )

28. Three Failure Regions & Three Sub-domains E (Θ) = {(1- dA)* (1- dB)* (1- dC)*(q)} + {(1-dA)* (1-dB)* (dC)*(q –F3/M)} + {(1-dA)* (dB)* (1-dC)*(q –F2/M)} + {(1-dA)* (dB)* (dC)*(q –F2/M–F3/M)} + {(dA)* (1-dB)* (1-dC)*(q –F1/M)} + {(dA)* (1-dB)* (dC)*(q –F1/M–F3/M )} + {(dA)* (dB)* (1-dC)*(q –F1/M–F2/M)} + {(dA)* (dB)* (dC)*(q –F1/M–F2/M–F3/M)} = q[ 1 – dCF3/ M*q – dBF2/M*q – dAF1/M*q] = q – { dAF1/M + dBF2/M + dCF3/M}

29. Simulator • Based on the above mentioned & verified technique a simulator was designed for empirical calculation of E (Θ) • Coding Language : Java • Graphical User Interface : Java Swings

30. Simulator screen shotsStep I: Specify the total number of Sub-Domains & Failure Regions

31. Step II: Specify the number of points per Sub-Domain, Failure Region D, 30 C, 30 A, 20 B, 20

32. The result got from running the simulator is similar to the one calculated theoretically (above). This proves validity of the Simulator Design.

33. Summary • Value of E (Θ) for software with multiple failure regions spread across multiple sub-domains can be calculated using: E (Θ) = q – ( dAF1/M + dBF2/M) E (Θ) = q – ( dAF1/M + dBF2/M + dCF3/M)

34. Future Work • The number of test-cases per sub-domain was limited to only 1 • We would like to generalize the present formula further by considering the effect of ‘n’ (multiple) test cases per sub-domain. • No generalization doesn’t merely involve putting ‘T’ as a exponential term in the equation ( have tried that )