Fault-Based Analysis: Improving IV&V Through Requirements Risk Reduction '02

1. Fault-Based Analysis: Improving IV&V Through Requirements Risk Reduction '02 Jane Hayes Rama Bireddy D.N. American SAIC Department of Computer Science University of Kentucky

2. Outline • Research Objective • Research Approach • Progress to Date • Current Plans • Future Work

3. Research Objective • To improve how we focus resources for IV&V of Critical/Catastrophic High-Risk (CCHR) software functions, • we use a fault-based analysis method comprised of: • a requirements fault taxonomy (Phase I) • a method for extending taxonomies (Phase I) • a taxonomy of IV&V techniques (Phase I and II) • what faults the techniques can detect (Phase II+) • a fault-based risk assessment per Class and project (Phase I+) • a cost-benefit analysis of technique effectiveness (validated) (Phase II+)

4. Research Approach (Phase I) • Task 1 – Select a Known Fault Taxonomy • Task 2, 3 – PMR 2,3 (Presentation and Milestone Meeting) • Task 4 – Examine NASA-specific requirements faults • Task 5 – Build a list of IV&V techniques • Task 6 – Adopt or build a method for extending the taxonomy • Task 7 –Implement the method to extend the fault taxonomy • Task 8,9 - Year-end report and presentation (also PMR 4)

5. Progress to Date • NUREG/CR-6316 basis of general taxonomy • Literature survey (55 references) resulted in threeadditions to taxonomy • Review of defect data for 3 NASA projects resulted in two additions and re-organization of taxonomy

6. General Taxonomy Taxonomy figure

7. NASA Requirement Faults • Have proven difficult to obtain • Level of detail varies greatly • Have thus far received and examined • IV&V “comments” on requirement problems for 3 projects • Project fault reports related to requirements for 1 project • Data very useful, resulted in several changes to fault taxonomy and taxonomy extension/tailoring processes

8. Task 6 • Process for extending fault taxonomy split into two parts: Process A and Process B • Process A - activities to develop a Class-specific taxonomy • Outputs of Process A are inputs to Process B • Process B – activities to develop a project-specific taxonomy

9. Processes for Extending Fault Taxonomies High level process

10. Class-Specific Taxonomy Process Process

11. Project-Specific Taxonomy Process Process

12. Current Plans • Obtain requirement-related fault reports for additional NASA projects • Perform Process A (Class-specific) for classes for which we have data • Complete list of IV&V techniques • Continue sharing information and soliciting feedback from ST-5 project • Prepare final report

13. Future Work (Phase II) • Build taxonomy of IV&V techniques • Survey literature and determine what techniques have been shown to detect certain types of requirement faults • Gather expert opinion to fill in gaps of the technique-to-fault matrix • Design experiments to validate some of the technique-to-fault mappings • Provide resulting information in an Advanced Risk Reduction Tool (ARRT) friendly format

14. Backup

15. How does this differ from ODC? • ODC uses a fixed set of trigger and defect types • Our emphasis is on building tailored taxonomies • We use historical information about Classes of related projects • ODC classification strives to be independent of the specifics of a product or organization

16. How does this differ from ODC? (cont’d) • ODC defect types don’t map well to requirements (function, I/f, checking, assignment, algorithm, etc.) • We integrate risk analysis in our taxonomy building process • Our long-term goals include validated cost-benefit information for fault-technique pairs