David Marzo Operations Research Analyst, Weapon Systems Cost Analysis Division

1. Joint Strike Fighter: Case Study of Analyzing the Costs of a Major Weapon System David Marzo Operations Research Analyst, Weapon Systems Cost Analysis Division U.S. Department of Defense OSD Cost Assessment and Program Evaluation June 2011 david.marzo@osd.mil (703) 697-2452

2. Agenda • OSD CA Overview • Recurring Challenges & Considerations • Joint Strike Fighter Example to illustrate CAPE estimating proclivities

3. OSD CAPE Overview • Analytical arm within the Office of the Secretary of Defense • Cost Assessment (CA): Mandatory independent cost estimates and internal discretionary reviews • Program Evaluation (PE): Broader portfolio review & oversight • CA also known as the CAIG or Cost Analysis Improvement CAIG • Established in 1972 • Bolstered through a series of regulations & acts, most recently the 2009 Weapon Systems Acquisition Reform Act

4. OSD CA Today • Staff size: 30 Full-Time Equivalents (FTEs) • 25 analysts available for CAPE projects -- 16 analysts qualified to lead CAPE projects • About 55-60 CAPE estimates per year* • Average time commitment per cost estimate • 6 calendar months • 8 to 10 FTE-months of effort • Perform full lifecycle cost estimates (Development, procurement, military construction, operating & support) * Includes special studies and Nunn-McCurdy support

5. Agenda • OSD CA Overview • Recurring Challenges & Considerations • Joint Strike Fighter Example to illustrate CAPE estimating proclivities

6. Cost Estimating Challenges • Program definition uncertainty • Fallacy of Detail • “Black Box” Models • Optimism • Actuals: The “Gold Standard”

7. Program Uncertainty Over Time Pre Systems Acquisition Systems Acquisition Sustainment Pre Systems Acquisition Concept Refinement Concept Refinement Engineering & Manufacturing Development Production & Deployment Operations & Support Cost, Technical, and Programmatic Uncertainty A B C IOC FOC Disposal ASR SRR PDR TRR OTRR PCR IBR Technical Reviews System Specification/ CDD SFR Allocated Baseline CDR FRR SVR/PRR ECPR Preferred System Concept System Functional Baseline Product Baseline Product Baseline Parametric/ Analogy Parametric/ Analogy Parametric/ Analogy Engineering Build-Up (PO & Contractor) Extrapolation of Actuals Analogy Parametric Engineering Build-Up (PO & ktr) Extrapolation of Actuals Analogy Parametric Engineering Build-Up (PO& Ktr) Estimating Techniques Program definition uncertainty lessens over time FOR OFFICIAL USE ONLY

8. Fallacy of Detail • Level of aggregation is probably one of the most important decisions made in an estimate • Greater complexity/detail is often seen by the authors as an index of estimate quality (after all, it reflects great effort) • Subject to the erosion of accuracy for the sake of precision • Observation: Cost Analysis—as a discipline—inevitably relies on the proposition of correlation • Highly correlated elements MUST be consolidated • Failure to consolidate correlated elements will provide erroneous results • This is the fundamental flaw in cost risk analysis as generally practiced • Consolidating correlated cost elements simplifies the estimating effort, and focuses analytic attention on the fundamental drivers of cost • Note that the availability of lower level data does not compel its use Better to have an estimate that is “Approximately Right” rather than “Precisely Wrong”

9. “Black Box” Models and Evidence • First requirement of an estimate is that it be trusted • Estimator is asking senior managers to bet their careers . . . • Must explain—in quite specific terms—the chain that runs from data/evidence, through models/methods to final estimate • Proprietary “closed” models are inherently problematic • Many commercial forms • One is left with showing inputs, black box, and estimates • The best one can say of credibility is the “trust” in the trademark • As an estimate consumer, one inevitably wonders • How would I know if X model is appropriate for this application? • Have the inputs been gamed, particularly through “calibration”? Other than “trust me”, what do I have to believe to have confidence in this black box estimate?

10. Optimism Delusions of Success By Dan Lavallo and Daniel Kahnemon, Harvard Business Review, July 2003 Key Points from the Article • When planning major initiatives • Routinely exaggerate benefits and discount costs when projecting risky project • Sets up project for failure; psychologists “planning fallacy” -- spin success scenarios and overlook potential for mistakes • Optimism traced to cognitive biases and organizational pressures • People highly optimistic most of the time – exaggerate talents and degree of control; attribute negative consequences to external factors • Competition ($, time) for new projects – incentive to accentuate the positive • Anchoring magnifies optimism; initial plan accentuates positive, skews subsequent analysis • Most pronounced for initiatives companies have never attempted before • Temper with “outside view” • Supplements traditional forecasting w/ statistical analysis of analogous efforts – reality check on initiative inside view • Don’t remove optimism, but temper effects -- balance between optimism and realism - goals (motivate) and forecasts (decide whether to make commitment in the first place)

11. Actuals: The Gold Standard • Actuals have one sterling advantage: they reflect all “unknown/unknown” problems of achieving a capability • Actuals-based estimates are easy to explain • Two distinct cases of actuals estimating: • Early Actuals on a specific program to extrapolate future lots • Actuals from other programs, often adjusted for content/scope, to forecast costs of new program Even if you don’t use actuals for a particular element of cost, presenting relevant ones helps to calibrate the discussion

12. Is Cost A Self-Fulfilling Prophecy? • Common Complaints • “CAPE Estimates are inherently negative and reinforce past mistakes” • “I don’t need all of this extra money—if you give it to me, the program will just spend it unnecessarily.” • “Cost is a self-fulfilling prophecy” • Thoughts • Programs can address known “mistakes” from past programs and not repeat them – but each program has its unique unknowns—the inevitable challenges of complex weapon systems • Trick is to find the dividing line between too much and not enough –the “thesis analogy” • Good management has an important role

13. Agenda • OSD CA Overview • Recurring Challenges & Considerations • Joint Strike Fighter Example to illustrate CAPE estimating proclivities

14. Joint Strike Fighter • What is the program status? • How is CAPE involved? • How do we approach estimating it? What are some of the unique estimating challenges? • How does the Department react to differing cost estimates? • What are the impacts of international procurements?

15. JSF – Program Status • Program underwent a “Nunn-McCurdy” cost breach in summer 2010 • Program originally based in 2002 • Projected unit cost >50% resulted in critical breach • Department of Defense must re-assess requirements, cost, status • Summer 2010: Department states no other alternative meets requirements within acceptable cost—proceed with JSF

16. JSF Program Status • Fall 2010 : Comprehensive Technical Baseline review resulted in a restructured development program • Block 3 Qualification Projected Complete in 2016 • CAPE position: reasonable development baseline established; focus on execution; challenges remain

17. CAPE Involvement • Departmental Internal Requests • 2008 Joint Estimating Team (“JET 1”) • 2009 Joint Estimating Team (“JET II”) • Statutory Requirements for Cost Estimates • 2010 Nunn McCurdy Cost Breach Review: Development and procurement • 2011 Milestone B Review (Ongoing): Development, procurement, operating and support

18. JSF Estimating Approach • Development • Function of schedule and personnel staffing levels; “time is money” • Remaining schedule is a function of test point burndown • Project actual costs forward; shaped by legacy history • Procurement • Pre Nunn McCurdy: Based on analogy to F/A-18, F-22; adjusted for weight and system complexity • Since Nunn McCurdy • Based on actuals from JSF lots; bounded by legacy history • Separately model each section of aircraft; and separate major contractors

19. Theoretical Example

20. Challenge of Modeling Commonality Past Future ? CTOL 3. Fundamental Difference? STOVL 2. Temporary Discontinuity? 1. 100% Commonality?

21. Data Example #1 SDD Production

22. Data Example #2 SDD Production

23. History as a Bound JSF Estimates vs. Legacy History Compare CAPE Estimates to analogous historical experience – Is our estimate within a logical range given legacy experience?

24. The Challenge of Concurrent Development and Production

25. Operating and Support Cost • Operating and support (O&S) cost • It will cost more to operate and maintain JSF than legacy fighter platforms; however, JSF is more capable • Significant drivers of O&S cost (e.g., Depot Level Reparables, engine/fuel consumption) are driven by inherent aircraft characteristics already designed in (e.g., stealth, complex mission systems) • Other significant drivers of O&S cost are driven by ops requirements (e.g., mission personnel & flight hours) The Department is continually examining the major drivers of sustainment costs and aims to capitalize on various opportunities to reduce costs

26. Annual “Steady State” Cost Example Acronyms: PAA: Primary Aircraft Authorized TAI: Total Aircraft Inventory DLR: Depot Level Reparables

27. Other Challenges of Estimation • Concurrency of phases—same contractor staff charging simultaneously to development, production, and sustainment—need to account for correctly • International buy profiles – international purchases move processes further down the curve; when US defers purchases, how do we model shifts in international profiles? • Evolution of production processes – what is built in house now may be outsourced later

28. Department Treatment of Cost Estimates • Department leadership has a choice against which estimates to baseline and budget against • “Fund to program estimate” or “Fund to CAPE estimate” • Each program is different • For JSF, this has been an evolutionary process— • For development, incremental movement over past 3 years towards funding to CAPE and Tech Baseline Review estimates • For procurement, Department has funded to CAPE estimates. There is still ambiguity, however, with respect to the effects of commonality and its impact on costs • Assessing risk and funding to the 50% estimates vs 80% estimates

29. Impact of International Participation • We estimate that the US gets about a 5% unit cost benefit from international participation • International participation does introduce complexities • Establishment of production capabilities in other countries • Shifting quantity profiles

30. Closing Comments • CAPE has responsibility for performing cost estimates in response to both US statute as well as internal Departmental requests • Our approach emphasizes: • Using a combination of historical analogous data & actual costs when available • Relatively simple but sound logic • JSF is a good example of our involvement and cost estimating methodology; we recognize that estimating JSF is a dynamic process informed by ongoing dialogue with the contractor and program office