Naval Center for Cost Analysis Brian Flynn PDI 2008

1. Naval Center for Cost AnalysisBrian FlynnPDI 2008 NATO Independent Cost Estimating & Capability Portfolio Analysis

2. Contents • Introduction • Background • NATO • Task Groups • Cost Analysis • Findings • Recommendations • Handbook • Exploratory Team • Road Ahead • ICEs • Portfolio Analysis

3. Vision Improve cost estimating in the international defense community on systems such as F-35 Joint Strike Fighter NATO Early Warning and Control System (AWACS) NATO Alliance Ground Surveillance System (AGS) Introduction

4. NATO/OTAN Organisation du Traite de l’Atlantique Nord “The Parties of NATO agree that an armed attack against one or more of them … shall be considered an attack against them all.” NATO Members

5. NATO Membership

6. NATO Expansion Military Highlights • 1954 … Soviet Union tries to join • 1994 … first military action; Bosnia • 1999 … Kosovo War; Yugoslavian leader Slobodan Milosevic captured • 2003 … NATO takes command of the International Security Assistance Force (ISAF) in Afghanistan

7. NATO Research • Task group on cost analysis created and approved by NATO’s Research and Technology Organization • Mission of RTO To meet the military needs of the Alliance; To provide advice to NATO and national decision makers. • “To support the development and effective use of national defence R&T and thus to maintain a technological lead within the Alliance;

8. RTO Role Research and Technology Board (57) Panel Members (373) Scientists and Engineers (3,500) RTA Headquarters (61)

9. Task Groups Every Task Group needs a TAP (Technical Activity Proposal) and a TOR (Terms of Reference) Panels Technical Teams

10. Cost Analysis Current NATO Specialist Team on Ship Costing 1991 (on-going) SAS-028; Cost Breakdown Structure for NATO 2003 NATO Cost Analysis Improvement SAS-069; Handbook of best practices. Lead: Mr. Marcel Smit SAS-054; Methods and Models for Life Cycle Costing. Lead: Mr. Marcel Smit Future SAS-076; ICEs and Portfolio Analysis Lead: NCCA (Brian Flynn) 2004 to 2006 AC/327 Working Group on Life-Cycle Costs

11. SAS-054 • Membership • NATO and Partnership for Peace (PfP) nations invited • Participants • Denmark, Germany, Norway, Switzerland (PfP), United Kingdom, France, The Netherlands, Sweden (PfP), Turkey, United States, OCCAR • Observers • Belgium, Italy, Greece, Georgia (PfP) Manages seven European armaments programs Organisation conjointe de coopération en matière d'armement

12. Partnership for Peace alliance aims to create trust between NATO and other European states and the former Soviet Union Created in 1994 23 members SAS-054 ██ Current members of PfP ██ members that have left PfP ██ PfP members that joined NATO in 1999 ██ PfP members that joined NATO in 2004

13. Objectives Understand NATO and PfP nations’ methods and models for cost analysis Promulgate best practices within NATO’s Phased Armaments Programming System (PAPS) SAS-054 Lead: Mr. Marcel Smit, The Netherlands Executive Secretary (Chief of Staff): Mr. Arthur Griffiths, United Kingdom

14. SAS-054 • Method • Each nation asked to map its acquisition process into PAPS • Then, best practices in cost analysis identified for each phase • U.S. mapping:

15. Data Collection Critical but difficult for all nations Availability of data often defines what methods and models can be applied Methods and Models Standard estimation tools like OLS are common Findings

16. Risk and Uncertainty Life cycle cost estimates are widely regarded as probability distributions Point estimate merely one observation Sensitivity analysis and use of SMEs common Findings

17. Presentation of Results Little standardization within or between nations Essential to convey the “right” information cogently Exchange of Information Significant barriers Difficulty in accessing each others’ websites and databases Findings “the basic problem,” from PARC

18. Life cycle cost estimates should: Be fully documented Be prepared by experienced personnel Include an affordability analysis Use cross-checks Life cycle cost models should: Be validated by a recognized testing process Data collection efforts should: Focus on cost, programmatic, technical, and performance information Within the entire NATO/PfP community To improve the accuracy, visibility, and availability of data useful to all Recommendations Breakdown barriers on information access

19. Risk and uncertainty analysis should be based on one or more generally accepted techniques such as Sensitivity analysis Monte Carlo simulation Garvey’s scenario-based analysis U.K.’s risk registers Risk and uncertainty analysis should start early With analytical justification of values Don’t leave to the 11th hour Recommendations Develop early, with each point estimate

20. Conduct Monte Carlo simulation along these lines Generate a baseline estimate that reflects uncertainty “Noise” or variance in estimating relationships Then include risk Delta between the means is cost risk Recommendations

21. Convey to decision makers the essential fact that cost estimates are stochastic List assumptions or describe scenarios Recommendations Display a range of estimates

22. Code of Practice for Life Cycle Costing Describes best practices for all phases of the life cycle Examples Handy booklet End date: Sep 2008 SAS-069 Lead: Mr. Marcel Smit, The Netherlands Executive Secretary (Chief of Staff): Mr. Arthur Griffiths, United Kingdom

23. Membership NATO Norway, United Kingdom, The Netherlands, Germany, United States, Greece, Poland, Canada, Turkey Partnership for Peace Switzerland and Sweden OCCAR Exploratory Team

24. Objectives Explore follow-on activities to SAS-054 and SAS-069 Methods and Models of Life Cycle Costing Brochure Options Demonstrate proof of concept Independent life-cycle cost estimates (ICEs) Improve data collection Perform capability portfolio analysis Focus on software development Exploratory Team Lead: Mr. Marcel SMIT, The Netherlands Chief of Staff: Mr. Arthur GRIFFITHS, United Kingdom

25. Recommendations Test guideline Generate two or three ICEs Over the life cycle Fully documented With risk and uncertainty analysis Identify best practices in capability portfolio analysis Emphasis on the role of life-cycle cost analysis Exploratory Team

26. Process of system selection Exploratory Team Identify Candidates (Importance, Size, Point in Life Cycle, Type of Platform) • Technical Activity Proposal • Ex-post testing • Fielded system • Description available • Test ICE against actual costs • Ex-ante estimation • New system • ICE to support decision making NH-90 NATO AGS Rotterdam Ships A400M FREMM No Approval Yes TAP = f(recommendations, results)

27. Leadership Study Director Dr. Brian FLYNN, NCCA, United States Chief of Staff Mr. Dave BAGGLEY, United Kingdom Senior Strategic Advisor Mr. Arthur GRIFFITHS, United Kingdom Participants (to date) NATO Norway, United Kingdom, The Netherlands, Germany, United States, Greece, Poland, Canada, Turkey, Italy, France Partnership for Peace Switzerland and Sweden OCCAR Time line 2008 to June 2011 SAS-076 Liaison with NATO Defense College; Colonel Massimo PICA, Italy

28. Independent Cost Estimates (ICEs) Ex-post testing Program description Cost Analysis Requirements and Assumptions (CARA) for NATO ships Test against actual costs after ICE Ex-ante estimation NATO Alliance Ground Surveillance System To support NATO and national decision making SAS-076 Rotterdam Class Ships

29. SAS-076 Cost analysis is a data-driven process Task Group will begin laying-in baseline (schedule, tasks, and assignments) at first meeting

30. Issue: scope of ICEs Notional example for NATO AGS Task Group will address early-on SAS-076 Potentially seminal work Command, Control, and Communications Infrastructure Costs

31. Capability portfolio analysis Best practices Two or three international workshops Subject matter experts from each nation E.g., Joint Staff, USD(AT&L), SOCOM HQ in the U.S. Emphasis on role played by life cycle cost analysis SAS-076 “In defense planning, capability is defined as the enduring ability to generate a desired operational outcome or effect, and is relative to the threat, physical environment, and contributions of coalition partners.” [Through Life Capability Management Conference, London, 2007]

32. SAS-076 Task Group will begin laying-in baseline (schedule, tasks, and assignments) at first meeting

33. Value Engender more informed decision making in the Alliance to support coalition warfighters Independent cost estimates First-ever for NATO Improved accuracy and visibility Test of guideline of best practices Capability portfolio analysis Examination of role of life-cyclecosts in addition to capabilities and risks Better initial, up-front decisions on allocation of limited Alliance resources SAS-076 Coalition forces in Afghanistan

34. Backup

35. Example of initial effort in executing capability portfolio analysis NCCA’s pilot in mine warfare SAS-076

36. MCM Portfolio

37. Weightings Identification and Ranking of Strategic Weightings Tasks Weightings -- Effectiveness -- Risk Tied to National Defense Strategy and Presidential Directives Design of Scoring • Strategy-to-systems model for MCM assets Operational to Strategic Tasks Tactical to Operational Tasks Systems to Tactical Tasks

38. Value of Systems Detection, Localization, Classification, and Identification of Sea Mines Numbers shown are ROIs ROI = Military Value/Cost Each bubble captures development, procurement, and ten years’ of O&S cost