1 / 29

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard.com pgb@csr.city.ac.uk Adelard, Drysdale Building, Northampton Square, London EC1V 0HB +44 20 7490 9450 www.adelard.com. Outline of Talk. Original reliability bound theory (same op. profile)

zaina
Download Presentation

Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard.com pgb@csr.city.ac.uk Adelard, Drysdale Building, Northampton Square, London EC1V 0HB +44 20 7490 9450 www.adelard.com

  2. Outline of Talk • Original reliability bound theory (same op. profile) • Extended theory (different operational profile) • Implications of the theory • Experimental evaluation

  3. Original Theory Input Domain Defect D 1 Operational 1 Observed profile (I) 2 D defect 2 failure frequency D 3 3

  4. Theory assumptions • the operational profile is invariant, i.e.s are constant over time • when a failure occurs the associated defect is immediately and perfectly corrected • removal of a defect does not affect the s of the remaining defects

  5. Basic idea Given some test interval t : • Defects with large s will be removed already • Defects with small s will remain - but have little affect on program reliability • So there must be an “worst case”  for a defect that maximises the program failure rate after t

  6. Worst-case bound • Original paper showed that, given the assumptions, max failure /unit time for a defect iis: i|t  1/et(where t is the test time) • So if there are N faults in the program the failure rate at time tis bounded by: |t  N/et

  7. Bound is independent of l 1 l =0.1 l =0.01 l =0.001 0.1 1/et 0.01 Probablity of failure 0.001  | t 0.0001 0.00001 1 10 100 1000 10000 t

  8. Refinement for discrete tests • For for a discrete sequence of T tests the result is: |T  N (T/T+1)T/(T+1) N/(eT) (conservative approx.) • So it is conservative to use original equation.

  9. Limitations • Assumes operational profile I is constant hence ls are constant • But we know that in practice the profile changes. • So the reliability bound does not apply if the operational profile changes • (e.g. from system test to actual use) • but will “settle back” in long term if new profile stable • New theory gives a means for “rescaling” the reliability bound for a different profile

  10. Additional assumptions • Each defect is localised to a single code “block” • The operational profile I can be characterised by the distribution of code block executions Q in the program {q(1), q(2), … } • The failure rate of defect in block, l(i)  q(i) • There is a constant probability of a fault existing in any line of executable code.

  11. Rescaling for known defect • For a defect i in code block j , the re-scaled bound would be: where q’(j) is the new execution rate and q(j) is the old execution rate.

  12. Probability of defect in block • We do not know which block contains defect i, but we assume that the chance of being in jis: L(j)/L where L(j) is the length of the code block, and L is the total length of the executable code.

  13. ¢ q ( j ) L ( j ) å × q ( j ) L Re-scaled bound • Taking the average over all blocks: • So the “scale factor” relative to the original bound is: • Also true if there are N faults rather than 1

  14. Theory predictions - Fair testing • If q  L of blocks “dominated” by decision branch,scale factor unchanged by any other profile • Applies to any acyclic graph, • And subgraphs with fixed iteration loops Segment j L(j)q’(j) q(j) L(j). q’(j) q(j) Root 0 10 1 1 10 Branch 1 10 0.1 0.9 90 90 0.9 0.1 10 Branch 2 Sum 110 110 S =Sum/L 1

  15. Unfair testing • Use of “unbalanced” test profile can be very sensitive to changes in profile • Factor can be less than 1 if under-tested blocks avoided, e.g. Q’={1,1,0} gives S = 0.19 q’(j) Segment j L q’/q q(j) L(j) Root 0 10 1 1 10 Branch 1 10 0.9 0.1 1.1 810 90 0.1 0.9 Branch 2 Sum 110 829 S =Sum/L 7.5

  16. Limits to fair test approach • Fair test apportionment does not work for variable loops, recursion and subroutines • Even if we identify a fair test profile, it may be infeasible to execute Decisions not independent (shared variable)

  17. Maximum scale factor • If we know max. possible execution rates for each block, can estimate a “maximum scale factor”: ( q(k) max / q(k) ) (L(k) / L) • Where k relates to a worst case “thread” through the graph. Hard to identify this thread, but easier to compute a more pessimistic factor: ( q(j) max / q(j) ) (L(j) / L) where j includes all blocks. • No knowledge of the new profile is needed

  18. ¢ L ( j ) q ( j ) å × + L q ( j ) x ( j ) / T Including module tests • Can combine module tests and system tests, composite scale factor is: where x(j) are the total executions under module testing • Module tests can “fill in” uncovered segments that would make the test profile “unbalanced”

  19. Experimental evaluation • Use programs with known set of defects • PODS • simple reactor trip application (<1000 code lines) • simple structure, fixed loops • PREPRO • ~ 10 000 code lines • parses input description file of indefinite length • recursive - max execution unknown • Similar results - will only discuss PODS here

  20. PODS evaluation • Measure Q for different test profiles • Uniform, Normal, Inverse normal - “bathtub” • Measure defect failure rates l(i) under all profiles • Predict residual failure rate:l(i) exp(-l(i)T) • Compute failure rate for new profile:l’(i) exp(-l(i)T) • Compare with scaled bound: (L(j)/L)(q’(j)/q(j))N/eT

  21. Variation in q(j)

  22. Predicted scale factors Operational profile Test profile uniform inv-normal normal uniform 1 1.2 0.9 inv-normal 3.2 1 6.2 normal 115 346 1 • Note the predicted reduction in bound

  23. Maximum scale factor Test profileMaxscale-up factor uniform 6.6 inv-normal 10.0 normal 1059 • 2-5 times worst than bound with a known profile • Can be over-pessimistic • But could indicate relative sensitivity to change

  24. 1 0.1 0.01 Operation (uniform) Mean 0.001 Fails/test 0.0001 (normal) 0.00001 0.000001 10 100 1000 10000 100000 Tests “Unfair” Normal test profile Max bound Scaled bound N/et bound

  25. 1 0.1 0.01 Test Mean (uniform) 0.001 Fails/test 0.0001 Operation (normal) 0.00001 0.000001 10 100 1000 10000 100000 Tests “Fairer” Uniform test profile Max bound N/et bound Scaled bound

  26. PREPRO • Similar results • changes in failure rates are within the scaled bounds • But could not compute a maximum bound • program is recursive • so no upper bound on the execution of program code blocks

  27. Summary Theory suggests: • Can rescale bound (knowing Q and Q’) • Can include module test execution information • Can compute max scale up (knowing Q and Qmax) • For some program structures can identify a totally "fair" test profile - bound insensitive to change The experimental evaluations appear to be consistent with the predictions of the theory

  28. Conclusions • Could affect approach to testing: • “fairer” test profiles rather than realistic profiles • integrated module and system test strategy • Could improve reliability bound prediction for new environment • Could assess sensitivity to profile change • e.g. by computing maximum scale factor • But based on quite strong assumptions, need to: • validate assumptions • assess impact of assumption violation • evaluate on more examples

More Related