Return on Investment of Independent Verification and Validation: Indirect Benefits

Return on Investment of Independent Verification and Validation: Indirect Benefits James B. Dabney, Gary Barber, Don Ohi, Kurt Woodham Software Assurance Symposium 29 July 2003

Overview • Approach • Overall Results • Top Four Benefits • Conclusions

Study Approach • Benefits of Value • Quantifiable Characteristics • Quantification Methods

Benefits of Value • Identify as many benefits as possible • Determine whether each benefit can be substantiated by tangible data • Assess quantifiability of each benefit • Identify quantification • Published data • IV&V and project artifacts

Quantifiable Characteristics • Score benefits • Develop method to convert quantifiable benefits into consistent measure (EPM, dollars)

Quantification Methods • Identify quantification procedure for each benefit • Unique method for each benefit • Refine and test

Results – Candidate Benefit List • Identified 84 candidate benefits • Refined list to 26 quantifiable benefits • Scored (High=3, Medium = 2, Low = 1) • Universality (weight = 1) • Practicality (weight = 3) • Potential value (weight = 4) • Credibility (weight = 5)

Top Four Indirect Benefits • Improved testing • Reduced high-criticality errors • Requirements clarification • Reduced leakage to operations

Improved Testing • Direct benefit is correction of test case deficiency • Indirect benefit is error identified as consequence of corrected test case • Computing return • Identify IV&V issues resulting in test case change • Correlate issues with defects found in testing • Incorporate results into direct ROI computation • Apply high-criticality and phase leakagemethodology where appropriate

Improved Testing Example • Mission-critical ground software system • IV&V identified 61 test deficiencies • One added test case identified flaw in redundancy management state machine • Flaw would have prevented detection of serious system fault • Direct ROI return for this error is small, indirect large using high-criticality methodology

Reduced High-Criticality Errors • CARA focuses IV&V on high-criticality portions of system • Removing high-criticality defects has benefits beyond cost-to-fix (direct ROI) • Value of mission • Value of vehicle • Crew and control center time during mission • Compute using expected value based on estimated probabilities of occurrence

Reduced High-Criticality Errors Example • Incorrect handling of thermal control system failure • Disable primary GN&C computer • Fault not recognized – backup GN&C stays off line • Vehicle placed in uncontrolled state • Three undesirable operational outcomes • Loss of vehicle at cost of $40B, 0.00001 probability • Loss of mission at cost of $4B, 0.00001 probability • Failure to annunciate fault, $4M, 0.01 probability • Expected value = $480,000

Requirements Clarification • Requirements ambiguities • Wording open to misinterpretation • Conflicts between different representations of same requirement • Benefits of removing • Reduced development cost resulting from reduced probability of implementation errors • Reduced maintenance cost due to increase in understandability

Reduced Development Cost Computation • Increment to effective SLOC for requirements ambiguity found in requirements phase is where kw is function point weight, kL function point to SLOC factor, pi probability of error of type i resulting from requirements ambiguity, ks is SLOC reduction factor, di is ambiguous requirement removal rate phase i

Reduced Maintenance Cost Computation • COCOMO-II reuse model includes term SU for understandability • Estimate increment to SU as • FPA is function points associated with ambiguous requirements • FPT total function points • Increment to ESLOC (effective SLOC due to reuse)

Reduced Error Leakage to Operations • Direct ROI model assumes no defect leaks to operations • Experience indicates a significant number of errors will manifest in integration testing and operations • Expand direct ROI leakage model to include integration and operations • Requires adjustment to direct ROI coefficients

Conclusions • Identified and rank-ordered 26 unique indirect benefits of IV&V • Analyzed top four benefits in detail • Improved testing • Reduced high-criticality errors • Requirements clarification • Reduced leakage to operations • Direct and indirect benefits form comprehensive ROI model and basis of predictive ROI model

Return on Investment of Independent Verification and Validation: Indirect Benefits