Evaluating Testing Methods by Delivered Reliability

1. Section 3.4 Evaluating Testing Methods by Delivered Reliability Frankl, Hamlet, Littlewood, Strigini IEEE TOSE Aug98

2. 841f07frankl7sep13 3.2 SFR, w/o subdomains • d = StinF V(t) • P(Q=0) = 1-(1-d)T • P(Q=q) = (1-d)T • E(Q) = 0* P(Q=0) +q* P(Q=q) • = q(1-d)T

3. 3.3 single failure region, with sub • Debug with n subdomains • E(Q) = qP(1-di)T/n • Operational • E(Q) = q(1-q)T

4. Assumptions: multiple failure model

5. Multiple Failure,debug w/o sub • E(Q) = Sqi(1-di)T • E(Q) = Sqi(1-qi)T

6. Example 1 – assume 100 tests in domain

7. Example 2 – Debug w/o subdomains? assume disjoint

8. Example 3 – subdomain testing

9. Example 3 – Solve empirically

10. Empirical Tool • Abstract domain to integers 1-100 • Abstract failure sets into number ranges • Run N (upto 5K) test tests • Each test set may have subdomains or n random tests • Determine frequency of each detect/notdetect combination • Input q of each detect/notdetect combination • Calculate E(Q)

11. Example Code for(i=1;i<MAX;i++){ answer = 0; int j; for (j=1;j<6;j++){ test = 1+ rand()%80; if (test <= 10) answer = answer + 1; if (test > 20 && test <= 25 ) { answer = answer + 2; error[2] = 1; } if ( test > 30 && test <=35 && error[2] == 0) answer = answer + 2; if (test > 35 && test <= 36){answer = answer + 4; error[4] = 1; } if (test > 36 && test <= 37) answer = answer + 8; if (test> 30 && test <= 32 && error[4] == 0) answer = answer + 4; if (test > 60 && test <= 62 && error[4] == 0) answer = answer + 4; if (test > 62 && test <= 63) answer = answer + 16; } found[answer]++;

12. For Tuesday, Sep 18 • Read rest of article • Build tool • Do experiments for subdomain testing with multiple faults.