Independent Verification and Validation Current Status, Challenges, and Research Opportunities

1. Independent Verification and ValidationCurrent Status, Challenges, and Research Opportunities Dan McCaugherty IV&V Program Manager Titan Systems Corporation 5 September, 2001

2. Agenda • Where we are today • Developer challenges observed by IV&V • IV&V challenges • Opportunities for R&D

3. Where we are today

4. IV&V Gaining Acceptance • IV&V is gaining acceptance as a mechanism for mitigating software risk • IV&V is naturally evolving to also include programmatic assessments such as cost/schedule performance analysis • Recent NASA IV&V policy presents an opportunity for wider application and understanding of IV&V

5. IT Project Success Factors • Gartner Institute study identified the following essential areas as critical to IT project success1 • Risk Management • Communications • Resource Management • Lessons Learned • Additional areas based on professional experience • Project planning • Requirements!!! • Projected that implementation of sound project management approach emphasizing these key areas can reduce number of failed projects by 30%1 1GartnerGroup

6. IT Project Success Rates • Odds for Success1&2 • > 80% of IT projects are late, over budget, lack essential functionality, or are never delivered • 28% of IT projects just flat fail representing more than $81 Billion in losses • Of 15,000 data migration projects started in 1999, as many as 88 percent either overran or failed • Leading causes for project failure1 • poor project planning • failure to manage scope and change • failure to manage risk 1GartnerGroup 2The Standish Group

7. Developer Challenges Observed by IV&V

8. Incremental Delivery Definition • “Build a Little, Test a Little” approach can avoid major development failures, but creates other challenges, such as: • Maintaining end state specifications concurrent with increment specifications • Implementing the right requirements in the right builds to meet testable increment specific capabilities • Understanding the cost/schedule effects of pushing requirements into later builds

9. Early Definition of Interfaces and Key Architectural Concepts • Late interface or architecture changes or problems create significant re-work or are unmanageable • Distributed computing stresses system engineering efforts for early definition of • API’s, remote procedures, and reused objects • Redundancy Management, • Fault Detection, Isolation, and Recovery • Exception handling

10. Use of New Technologies • A great way to attract and keep highly skilled professionals • Majority of focus on what the technology can do, not enough focus on what it can not do or was never designed to do • Limited skill base for new technologies • If COTS, may not have future support

11. Measuring Progress or Quality • Measurable definitions for “Done” are needed. • Metrics efforts and collection mechanisms need to be part of the process from day one • Optimism (a good thing) must be tempered with facts

12. IV&V Challenges

13. IV&V Technical Challenges • Adapting IV&V methods to developer environments • Understanding system dynamic behavior • Convincing a project that the process is too risky • Validating Non-Development-Items • Reverse engineering incompletely specified systems

14. IV&V Managerial and Business Challenges • Demonstrating that selected IV&V tasks and scope are appropriate for the Project’s acceptable levels of risk • Appropriate IV&V integration into project • Demonstrating IV&V ROI • Accounting for IV&V data needs within developer contracts.

15. Opportunities for R&D

16. Opportunities for R&D • Research IV&V ROI • Past case study3 shows 1.25 < IV&V ROI < 1.82 • Recommend a study of many projects with varying amounts and types of IV&V • Correlate overall project success to IV&V effort types • Continue development of code reverse engineering tools and interface V&V tools • Software and Interface Analysis Tool (SIAT) • Interface Management System (IMS) • Develop Reconfigurable engineering simulation for Space Craft • Improve early “validation” analyses 3Rogers

17. Opportunities for R&D • Develop a guidebook identifying sets of IV&V tasks for certain types of systems and acceptable levels of mission risk. • Research history of IV&V method application and results • Develop a guidebook for developer managers on how to realize benefits of IV&V. • Contracts and agreements • Integration of IV&V into different types of projects • Reporting and issue resolution

18. Summary

19. Summary • There are many software challenges/risks out there • Many can be managed through use of accepted engineering practices • IV&V can help mitigate risk if used/applied correctly • We need to better quantify the effects of IV&V and the ROI • Recommended areas of R&D • Research IV&V ROI, effects of IV&V use • Develop tools that are needed today • Develop guidance for future IV&V'ers and IV&V users.