Software Assurance Symposium 30 July ~ 1 August 2003 Prepared by: Tom Robinson

1. NAVAL AVIATION SYSTEMS TEAM TEAM NASA Office of Safety& Mission AssuranceIV&V Cost Estimation – A Joint NASA & US Navy Collaboration to Model & Automate the Process Software Assurance Symposium 30 July ~ 1 August 2003 Prepared by: Tom Robinson Research & Engineering Dept. Naval Air Depot Jacksonville, FL RobinsonTE@navair.navy.mil Software Assurance Research Program FY-2003

2. IV&V Cost Estimation – A Joint NASA & US Navy Collaboration • to Model and Automate the Process (CSIP2002-46) • The U.S. Navy has joined with National Aeronautics and Space Administration (NASA) in a collaborative effort to model and automate the process of cost estimation for Independent Verification & Validation (IV&V) of software. • The Naval Air Systems Command developed an easy-to-use cost tool for system software and hardware integration testing that uses a bottoms-up work breakdown structure (WBS), combined with statistical regression cost analysis, that yields cost estimates with a high degree of confidence. • The thrust of this Initiative is to develop similar tools for use by NASA on IV&V projects, and ultimately to improve the speed, accuracy, and consistency of NASA software IV&V cost estimating. NAVAL AVIATION SYSTEMS TEAM TEAM Initiative Description CSIP2003-456

3. Gather, Compile, and Analyze Data detailing Existing NASA Software IV&V Activities. • Establish / Validate a Benchmark with Processes and Procedures for Software IV&V in use by Industry, Academia, and other Government Agencies. • Refine NASA Software IV&V Program Management Processes, Interfaces and Tools necessary to Improve the Speed and Accuracy of Software IV&V Program Scheduling, Risk Assessment, and Cost Estimating. • Develop, prototype, and validate new NASA Software IV&V Program Cost Estimating Tools. • Improve and Evolve NASA Software IV&V Business Processes and Techniques. NAVAL AVIATION SYSTEMS TEAM TEAM Goals / Objectives CSIP2003-456

4. - Item Sizes, etc - Labor Rates - Timeline Info Standard Contract - IV&V Criteria A B C D E F ICE Tool IV&V Should-Cost Estimator NAVAL AVIATION SYSTEMS Contractor Criticality Analysis and Risk Assessment Spread by FY TEAM TEAM All S/W Items (Items for IV&V in Red) BCWS Regular Consistent Reports ‘Earned Value’ ACWP BCWP Cost & Schedule Technical Metrics Project Tracking Process Improvement All data above contained and configured/controlled in a Common Data Depository Possible ICE Tools Relationship within a Business Model CSIP2003-456

5. Established a Joint IV&V Cost Estimating (ICE) Team comprised of Personnel from NASA, the Naval Air Systems Command, Software Industry, and Software Engineering Academia • Developed & Delivered IV&V Effort Man-hour Tool (EMT - ProLogic) • Developed the Prototype IV&V Cost Estimating Tool (‘ICE Tool’) • Compiled & Distributed NASA Software IV&V Activity / Task Matrix, • corresponding Dictionary, and Standard ‘Mission Class’ Criteria Tables • Established Initial IV&V Cost Estimating Work Breakdown Structure • Developed & Delivered the Pre-Production ICE Tool • Demo during this 2003 NASA OSMA SAS • Production ICE Tool in-work; Hosting data tables on the NASA IV&V Tools Lab network server – Production Delivery in October 2003 NAVAL AVIATION SYSTEMS TEAM TEAM Major Accomplishments CSIP2003-456

6. NAVAL AVIATION SYSTEMS TEAM TEAM IV&V Cost Estimating (ICE) Navigator Screen CSIP2003-456

7. 1 – ICE ‘Run’ ends with a prioritized summary list of S/W Items $4,563 K Probable Budget $ 574K Budget Shortfall (from Min.) $6,152 K Ideal Budget 25 % ID S/W Item 2 – S/W Item selected by ‘clicking’ Item in prioritized summary list NAVAL AVIATION SYSTEMS Available (4,563) 5.1.1.1.3 Telemetry Processing (TMP) Min.for Cert. (5,137) TEAM WBS Test IV&V Activity Req Des Code TEAM ü ü ü ü Ideal (6,152) 101 Criticality Analysis/Risk Assessment ü 65 ü Criticality Analysis 102 ü ü ü Hazard Analysis 103 ü ü ü 104 Risk Analysis 5 – Return to prioritized summary of all S/W Items ü ü ü 105 Independent Risk Assessment Save Adjustment Rationale Reason for change 4 – Return to IV&V Details for the S/W Item after saving reasons 3 – Any adjustment to any IV&V task activity for that S/W Item requests a reason be entered Viewing & Adjusting ICE Results Save Changes CSIP2003-456

8. NAVAL AVIATION SYSTEMS TEAM TEAM IV&V Cost Estimating (ICE) Tool Walk-Through CSIP2003-456

9. Continue to collect and refine IV&V Task Details (Man-hours, Scalars) and Mission Class criteria; Define & refine contract cost Reporting requirements to capture future IV&V costs • Survey Department of Defense programs for IV&V cost data to Assist in tuning / validating the NASA IV&V Cost Estimating Tools • Design and develop a statistical IV&V cost / forecast engine to Overlay the ICE Tool bottoms-up, engineering-based WBS cost estimate • Perform full-up Beta-testing of network-based ICE Tools; Analyze & Report findings; Develop “Tuning Utilities” and maintenance ‘Assistant’ • Initiate Independent testing of IV&V Cost Estimating Tools; Analyze Test results; Update / improve tools as required NAVAL AVIATION SYSTEMS TEAM TEAM Plans for Fiscal Year - 2004 … Transition! CSIP2003-456

10. Tom Robinson (“Robbie”) NAVAIR Research & Engineering (JAX) RobinsonTE@navair.navy.mil John Marinaro NASA IV&V Facility John.Marinaro@nasa.gov Christina Moats NASA IV&V Facility Christina.D.Moats@nasa.gov Ken McGill NASA IV&V Facility Kenneth.McGill@nasa.gov Tim Menzies, PhD WVU SE Research Chair Tim@Menzies.com Hugh Smith The M & T Company Hugh.Smith@cdicorp.com Dan McCaugherty ProLogic Inc. DanM@prologic-inc.com Josiah Devasirvatham Science Applications International Corp Josiah.Devasirvatham@saic.com Tom Marshall Titan System Corp (Averstar Group) Tom.Marshall@titansystemscorp.com Rich D’Alessio The M & T Company Rich.Dalessio@cdicorp.com … You ? Volunteers … NAVAL AVIATION SYSTEMS TEAM TEAM The ‘ICE’ Team CSIP2003-456