Content

1. Software Independent Verification and Validation(IV&V)NASA IV&V FacilityFairmont, West VirginiaJudith N. BrunerActing Director304-367-8202judith.n.bruner.1@gsfc.nasa.gov

2. Content • Why are we discussing IV&V? • What is IV&V? • How is IV&V done? • IV&V process • Why perform IV&V? • Summary • Points of Contact

3. Why are we discussing IV&V?

4. Setting the Stage In the 90s, the Commanding General of theArmy’s Operational Test and Evaluation Agency noted that 90 percent of systems that were not ready for scheduled operational tests had been delayed by immature software.

5. Software “Chaos” The Standish Group examined 8,380 Software Projects. 53% “Challenged” 16% Successful - In Budget Over budget - by 189% - On Time - Late by 222% - Meets Requirements - Missing 39% of Capabilities - User involved 31% Cancelled Note: For Large Companies - 9% were Successful - Development - 61.5% Challenged - Over budget by 178% - Late by 230% - Missing 58% of Capabilities - 29.5% were Cancelled

6. Error Densities Design & Implementation 23% Requirements Specification Installation & Commissioning 68% 9%

7. Increasing Cost of Changes The cost to correct an software error multiplies during the development lifecycle. Cost scale factor (Normalized to Requirements Phase)

8. What is IV&V?

9. Independent Verification and Validation (IV&V) • Independent • Technical: IV&V prioritizes its own efforts • Managerial: Independent reporting route to Program Management • Financial: Budget is allocated by program and controlled at high level such that IV&V effectiveness is not compromised • Verification (Are we building the product right?) • The process of determining whether or not the products of a given phase of the software development cycle fulfill the requirements established during the previous phase • Is internally complete, consistent and correct enough to support next phase • Validation (Are we building the right product?) • The process of evaluating software throughout its development process to ensure compliance with software requirements. This process ensures: • Expected behavior when subjected to anticipated events • No unexpected behavior when subjected to unanticipated events • System performs to the customer’s expectations under all operational conditions

10. Independent Verification & Validation Software IV&V is a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle Adapted to characteristics of the target program

11. How is IV&V done?

12. Requirements Phase • System Reqts Analysis • S/W Reqts Analysis • Interface Analysis • Process Analysis • Technical Reviews & Audits Design Phase • Design Analysis • Interface Analysis • Test Program Analysis • Supportability Analysis • Process Analysis • Technical Reviews & Audits Code Phase • Code Analysis • Test Program Analysis • Supportability Analysis • Process Analysis • Technical Reviews & Audits Test Phase • Test Program Analysis • Independent Test • Supportability Analysis • Technical Reviews & Audits Verify Verify Verify Validate IV&V Activities Throughout Lifecycle

13. IV&V Life Cycle Functions • IV&V Process provides tools and analysis procedures appropriate to each phase of the software development life cycle: • Formulation Phase: • Is development process sound, repeatable, and managed? • Requirements Phase: • Verify that system and software requirements are correct, complete, traceable and testable • Analyze system-level requirements: Are test plans and acceptance criteria sufficient to validate system requirements and operational needs? • Are testing methods sufficient to verify and validate software requirements? • Are the correct software development, management, and support processes in place? • Design Phase: • Does the design support the requirements? • Are test plans and test environments sufficient to verify and validate software and operational requirements? • Does the design have any characteristics that will cause it to fail under operational scenarios? What solutions are appropriate?

14. IV&V Life Cycle Functions (cont.) • Typical IV&V functions by Software life-cycle phase (cont.): • Coding Phase: • Does the code reflect the design? • Is the code correct? • Verify that test cases trace to and cover software requirements and operational needs • Verify that software test cases, expected results, and evaluation criteria fully meet testing objectives • Analyze selected code unit test plans and results to verify full coverage of logic paths, range of input conditions, error handling, etc. • Test Phase: • Analyze correct dispositioning of software test anomalies • Validate software test results versus acceptance criteria • Verify tracing and successful completion of all software test objectives • Operational Phase: • Verify that regression tests are sufficient to identify adverse impacts of changes

15. IV&V Testing Involvement • IV&V identifies deficiencies in program’s test planning • Program changes their procedures to address deficiencies vice IV&V independently test • IV&V may independently test highly critical software using an IV&V testbed • Whitebox • Stress • Endurance • Limit • Developer motivated to show software works • IV&V attempts to break software

16. IV&V Process

17. Defines IV&V scope and objectives Program Planning Requirements Design agreement CARA results IV&V proposal information Developer Code Test IV&V Planning Execute Assessment Normal Introduce issues at lowest level Allow developer time to respond Issue resolved Exception Introduce issues at lowest level Allow developer time to respond If no resolution, take issue to program IV&V Process Integrates IV&V into program Provides IV&V funding Resolves Exception issues Reflects IV&V in program mgmt plan Agrees to data transfer plan Reflects agreement in subcontracts IV&V in phase with development

18. Specifications Program goals Dev plans/schedules Estimate resource requirements Criticality Analysis and Risk Assessment IV&V Plan Yes Acceptable? No Revise scope: - Breadth vs depth - Exceptions IV&V Scope • Scope is determined so as to minimize the risk within the Program’s IV&V budget. Effort is based on: • Criticality and risk of system functions performed/managed by software • Budget limitations Program’s IV&V budget

19. For each Software Function: Set IV&V Analysis Level (IAL) Thresholds IAL CARA Score None: 1 < CARA < 2 Limited (L): 2 < CARA < 5 Focused (F): 5 < CARA < 8 Comprehensive (C): 8 < CARA < 12 Criticality: Rating Category Performance and Operations Catastrophic=4 Critical=3 Moderate=2 Low=1 Average Criticality Safety Cost/schedule Risk: CARA score Rating Category Complexity High=3 Moderate=2 Low=1 Technology Maturity Reqts Dfn & Stability Average Risk Testability Developer Experience CARA Scoring Methodology

20. CARA Criticality Sample Criticality Evaluation Criteria

21. CARA Risk Sample Risk Driver Criteria

22. Requirements Analysis IALs

23. Design Analysis IALs

24. Code Analysis IALs

25. Test Analysis IALs

26. Program processes Software schedules, development tracking, critical path analysis, configuration mgmt Ancillary developments Simulations, trainers, test environments Increased probability of success - Good processes allow early error identification and correction - Quality documentation enhances software maintenance IV&V Is Process As Well As Product Oriented

27. Program Identification of top risks Eval of Program Devel status Eval of Program Schedule status Status Reviews Status Reviews Status Reviews Status Reviews Phase complete analysis report IV&V Reqts Week Design Week Week Month Week IV&V is a program level “tool” to efficiently and effectively manage software development risk. IV&V Increases Program Awareness

28. Program Site S/W IV&V Facility Developer Site Pgm Mgmt IV&V Critical Mass of: - Analysts - Tools Eyes, Ears, Advocates, & Domain Experts (Validation) Developers IV&V Staffing Paradigm

29. Why perform IV&V?

30. Better Visibility into • Development • Better Decision Criteria • Second Source Technical • Alternative • Reduced maintenance cost • Reduced Frequency of • Operational Change • Better software/system • Performance • Higher Confidence in • Software Reliability • Compliance between • Specs & Code • Criteria for Program • Acceptance IV&V Benefits Technical Management

31. Summary

32. IV&V Key Points • IV&V works with the Project • Goal is project success • IV&V is an engineering discipline • IV&V processes are defined and tailored to the specific program • Mission, operations and systems knowledge is used to perform engineering analyses of system components • IV&V is most effective when started early • 70% of errors found in testing are traceable to problems in the requirements and design • IV&V works problems at the lowest possible level • Primarily work via established informal interfaces with the development organization - working groups, IPTs, etc. • Elevate issues only when necessary

33. IV&V Approach Efficiently Mitigates Risk • It is not necessary or feasible to perform all IV&V analyses on all software functions • IV&V resources allocated to reduce overall exposure to operational, development, and cost/schedule risks • Software functions with higher cirticality and development risk receive enhanced levels of analysis (‘CARA’ process) • Systems analyses performed to reduce costly interface and integration problems • Process analyses performed to verify ability to produce desired result relative to program plans, needs and goals • IV&V working interfaces promote timely problem resolution • Proactive participation on pertinent development teams • Emphasis on early identification of technical problems • Engineering recommendations provided to expedite solution development and implementation

34. Analyses Are Value Added and Complementary- Not Duplicative • Analyses performed from a systems perspective considering mission needs and system use, hazards and interfaces • Discipline experts assigned to perform analysis across all life cycle phases • Horizontal specialty skills are matrixed across IV&V functional teams to verify correct systems integration • Specialized tools and simulations perform complex analyses • IV&V testing activities complement developer testing enhancing overall software confidence • Developer testing focuses on demonstrating nominal behavior, IV&V testing activities try to break the software • Overall program integration, test and verification approach analyzed for completeness, integrity and effectiveness

35. Why use NASA IV&V Facility? Software IV&V, as practiced by the NASA Software IV&V Facility, is a well-defined, proven, systems engineering discipline designed to reduce the risk in major software developments.

36. NASA IV&V FacilityPoints of Contact • Judy Bruner • Acting Director • 304-367-8202 • judith.n.bruner.1@gsfc.nasa.gov • Bill Jackson • Deputy Director • 304-367-8215 • bill.jackson@ivv.nasa.gov