Integrated Cost & Schedule Risk Analysis Dynamic Integrated Cost Estimator (DICE) Model

1. Integrated Cost & Schedule Risk Analysis Dynamic Integrated Cost Estimator (DICE) Model Adelaide, Australia 28 June, 2011 This document contains Booz Allen Hamilton proprietary and confidential information and is intended solely for the use and information of the client to whom it is addressed. This data shall not be released to other contractors without written consent from Booz Allen Hamilton. Booz Allen Proprietary/Not for Distribution

2. Outline Introduction RealTime Analytics™ Overview of Joint Confidence Level Analysis NASA’s Joint Confidence Level Policy Dynamic Integrated Cost Estimator (DICE) JCL Analysis Tool

3. Attempt at Humor

4. Introduction • Whether assessing/analyzing NASA, DoD or Intelligence Community owned projects, the story is the same each time: Programs are increasingly experiencing growth above and beyond their initial cost and schedule estimates • This is not just a cosmetic problem: Cost and schedule growth delays capabilities and constraints the budgets of other programs causing a waterfall of instability • Studies have examined the reasons behind this growth reaching similar conclusions • Early program optimism leading to optimistic estimates • Insufficient cost and schedule reserves available to cover risk • Weak independent validation of cost and schedule • Recognizing this, many guides, including the GAO’s Cost Estimating Handbook, have included Risk Analysis as a required step in best practice cost estimating processes

5. RealTime Analytics™ • One of the challenges in performing cost and schedule analysis is the time required to run simulations • Risk analysis, required for all cost estimates by 2009 Weapons System Acquisition Reform Act of 2009 and GAO, requires the use of simulations • Run-times of minutes or hours prohibit most risk analyses models from being decision making tools as they can not be re-run during meetings • RealTime Analytics™ (RTA) is a collection of technologies, tools and methodologies allowing complex analytics to be performed far faster than using currently available methods • This presentation will focus on the Dynamic Integrated Cost Estimator (DICE) and the methodologies it addresses: • Joint Confidence Level Analysis (Integrated Cost & Schedule Risk Analysis) • RTA tools allow simulations to run up to 99.99% faster than comparable industry tools • Simulations formerly taking minutes or hours to run now finish in under 1 second • Allows decision makers to run an unlimited number of excursion scenarios without ever leaving the meeting room

6. Introduction to Joint Confidence Level Analysis • Decision makers are starting to recognize that there is rarely a relation between cost risk analysis results and the program’s schedule • This can lead to risk adjusted cost estimates that, if come to pass, will almost always imply associated schedule growth • From the other side, traditional schedule risk analysis typically leads to risk adjusted schedules that, if come to pass, will result in cost growth • Even when both of these analyses are performed on a program, they are typically done by disjoint groups under different sets of assumptions • Joint Cost & Schedule Risk Analysis is an attempt to integrate cost and schedule risk analysis in a way that produces meaningful, compatible results • This presentation will cover: • NASA’s JCL Policy • How JCLs are performed

7. What is Joint Confidence Level Analysis? • Historically, cost and schedule risk analyses are developed separately and their results are not compatible • This can lead to risk adjusted cost estimates that, if come to pass, will almost always imply associated schedule growth and vice versa • Joint Confidence Level Analysis combines cost and schedule risk analysis into a single, coherent output • Joint Confidence Level (JCL) analysis1 results in a bivariate distribution of final projected cost and schedule pairs • Joint Confidence Levels represent the probability of finishing at or under both cost and schedule • JCL Analysis allows a program to: • Defend budgetary and scheduling decisions • Prioritize risks and other threats based on their overall impact to the program and not simply their anticipated local impact • Develop more precise risk mitigation plans 70% Joint Confidence Level Curve Probability Schedule Cost 1Also known as Integrated Cost & Schedule Risk Analysis or Joint Cost & Schedule Risk Analysis

8. What is Joint Confidence Level Analysis? • The primary output from JCL analysis is the JCL scatter-plot • Each point on the scatter plot represents 1 iteration of a Monte Carlo simulation performed on the JCL model • JCL scatter-plot provides: • Joint Confidence Levels (e.g. For NASA Budgeting) • Relationship between cost and schedule (e.g. $ cost growth/days schedule growth)

9. JCL Analysis: NASA Policy & Air Force Research • JCL Analysis becoming more common in industry • In use by oil industry for years • Already a requirement for NASA projects per NPD 1000.5 • Air Force completing research on JCL Analysis1 • Several methods available for integrating cost & schedule; preferred method depends on program phase and available data • Parametric approaches: • Typically used phase A and before or by oversight groups • Divided into multiple regression and multivariate regression approaches • Build-up approaches: • Typically used following phase B by PMOs • Cost risk analysis and schedule risk analysis performed separately • Quality checks performed to make sure analyses are compatible • 1Joint Cost Schedule Model (JCSM): Recent AFCAA Efforts to Assess Integrated Cost and Schedule Analysis. Hogan, Greg (et al). SCEA 2011

10. JCL Analysis: Overview of Methods • Joint Risk Analysis: • Cost Loaded Schedule-Based Simulation (NASA Policy) • Multivariate Regression Joint Risk Analysis methods estimate cost and schedule simultaneously; JCL scatter plot produced directly from analysis • Cost Risk Analysis: • Inputs-Based Simulation • Outputs-Based Simulation • Scenario Based • Parametric Analysis When Joint Risk Analysis methods are not used, cost and schedule risk analyses must be combined using a Monte Carlo simulation Monte Carlo Simulation • Schedule Risk Analysis: • Parametric Analysis • Schedule-Based Simulation When risk analyses are combined, care must be taken to ensure results are compatible • 1Joint Cost Schedule Model (JCSM): Recent AFCAA Efforts to Assess Integrated Cost and Schedule Analysis. Hogan, Greg (et al). SCEA 2011

11. Joint Confidence Level Analysis Process at NASA • JCL analysis can be performed using existing resources; likely risk management, cost estimating, and scheduling personnel • The artifacts needed to perform a JCL analysis are: • A program schedule (IMS or analysis schedule) with uncertainty bounds on task durations • A quantified risk register (probabilities, cost and schedule impacts) where each risk is mapped to a task in the IMS • A cost estimate with uncertainty bounds that maps to the schedule • Creation of these artifacts requires communication between program’s cost estimating, scheduling and risk management staff

12. NASA Joint Confidence Level Analysis Policy • Joint Confidence Level Analysis has gained significant momentum recently • NASA is leading the way in the development of this methodology • NASA Policy Directive 1000.5 mandates that programs will be baselined at the “70 percent confidence level” using a “joint cost and schedule probability distribution”2 • NPD 1000.5 also stipulates that projects are funded at no less than 50% of the JCL or as approved by the decision authority, maintaining JCLs through the program lifecycle • The goal is to provide stronger assurance that NASA can meet cost and schedule targets3 • A recent GAO report cites NASA’s JCL policy as an effort “to provide transparency on the effects of funding changes on the probability of meeting cost and schedule commitments” 4 • NASA Cost Analysis Division (CAD) has developed a handbook to provide more information and guidance on this topic • Programs conducting JCL Analysis include James Webb Space Telescope and SOFIA • While the methodology has made substantial strides, the cost and schedule communities must overcome political and technical obstacles before full adoption 2 – NPD 1000.5 - http://www.hq.nasa.gov/office/codeq/doctree/10005.htm - January 15, 2009 3– JCL Status Report - http://www.nasa.gov/pdf/421542main_JCL%20Status%20Report-2010%20Feb.pdf – February 2010 4– GAO Report – “NASA – Assessments of Selected Large-Scale Projects” - http://www.gao.gov/new.items/d11239sp.pdf - March 2011

13. NASA JCL Model Prototype: DICE • In Fall 2010, NASA CAD commissioned the development of a JCL model prototype • The intention of this effort was to explore the value of producing a standard toolset for NASA programs conducting JCL analysis • Booz Allen created the Dynamic Integrated Cost Estimator (DICE) with a focus on streamlining the JCL process and decreasing simulation runtimes • Other key features of the DICE prototype development included: • Rapid schedule import from MS Project • Cost-Loading • Discrete Risk Analysis • JCL Scatter Plots and Iso-Curves • Benchmarking effort with other JCL tools • It is important to note that there are many tools that projects can use to develop JCLs, but DICE is optimized for this analysis

14. DICE facilitates Joint Confidence Level Analysis • DICE is an Adobe Flex-based tool for cutting-edge cost and schedule risk analysis • Includes modeling capability for producing build-up Joint Confidence Levels (JCLs) • Achieves industry-leading runtimes using Booz Allen’s RealTime Analytics

15. DICE – Benchmarking against Primavera Risk Analysis • Runtimes (3200-line schedule) • S-Curves (at 0.4 correlation) Total Project Cost Project End Date Schedule Cost • Key Points • Identical input parameters to ensure consistency in benchmarking • Importing risks from 3rd party template • Outputs <1% variation from Primavera • Correlation Project End Date

16. DICE Demo

17. Conclusion • Booz Allen’s recent innovations in simulation technology enable analysts to support decision making in near real-time • Decision makers can see the impact of changes to their program, real-time, without ever leaving the meeting room • There is no longer a limit on the number of excursions that can be run on an analysis • New Joint Confidence Level methods changes the way programs look at cost and schedule risk • Analyses are no longer performed and viewed separately, but rather are integrated and optimized using a standard tool • Decision makers have more insight into their program than ever before • Opening of communication lines between cost, schedule and risk management staff results in better program management

18. Points of Contact Colin Smith Associate Booz Allen Hamilton Inc. Suite 2100 230 Peachtree Street NW Atlanta, GA 30303 Tel (404) 658-8011 Smith_Colin@bah.com Brandon Herzog Consultant Booz Allen Hamilton Inc. 1530 Wilson Blvd., 10th Floor Arlington, VA 22209 Tel (703) 526-6040 Herzog_Paul@bah.com Eric Druker Associate Booz Allen Hamilton Inc. St. Louis, MO Tel (314) 368-5850 Druker_Eric@bah.com Booz | Allen | Hamilton Booz | Allen | Hamilton Booz | Allen | Hamilton Booz | Allen | Hamilton Graham Gilmer Associate Booz Allen Hamilton Inc. 1530 Wilson Blvd., 10th Floor Arlington, VA 22209 Tel (703) 526-2413 Gilmer_Graham@bah.com

19. DICE Functionality – Gantt Chart and Cost/Schedule Uncertainty • Organizes project tasks, costs, constraints, schedule interrelationships, and adds uncertainty to individual cost and schedule items

20. DICE Functionality – Cost-Loading • DICE can load schedules with time-dependent and time-independent costs, generating standard outputs such as JCL scatter plots and iso-curves

21. DICE Functionality – Fiscal Year Segmenting • DICE accounts for costs (and uncertainty) by fiscal year – aids budget planning

22. DICE Functionality – Run-time Features • DICE incorporates blanket correlation across the model, or specifies individual correlation between schedule tasks, time-independent costs, or risks • The model is optimized for Joint Confidence Level Analysis • Greatly increases understanding of how schedule growth impacts cost • Contains robust schedule logic functionality and discrete risk integration • DICE includes RealTime Analytics to enable industry-leading simulation runtimes • Reduces the time required to evaluate decisions and provides decision makers with on-the-spot analysis • Booz Allen’s RTA allows for quick initial runtimes and near-immediate re-runs • Interactive features provide intuitive user experience and rapid evaluation of alternatives • Enables comparison of multiple different scenarios or confidence levels of the same project • Builds off of existing tools (like MS Project, Excel) for seamless data integration, cost visualization, and navigation of risk analysis

23. RealTime Analytics: Excel Tool

24. RealTime Analytics Excel Tool™Introduction • RealTime Analytics™ Excel Tool • Excel add-in enabling fast running of simulations • Similar in capabilities to Crystal Ball and @Risk • Benefits: • Runtime savings of >99% vs. COTS tools • Analysis can be run without ever leaving the meeting room • Decreased simulation runtimes allows running of unlimited number of excursion scenarios • Not a numerical approximation approach such as Method of Moments, it simply runs simulations faster than existing tools • Uses: • Cost estimating/risk analysis/risk management • Insurance pricing/actuarial models • Portfolio optimization/trade-off analysis • Cash flow/profit analysis

25. RealTime Analytics Excel Tool:Benchmarking vs. Crystal Ball & @Risk Simulation Run Times File Size ~ 265 MB ~ ~ Crystal Ball 15 MB RTA Assumptions Comparison of Simulation Results • Simulations include mix of triangular, normal, beta, lognormal and uniform distributions with same parameters in each model • 444 correlated distributions with 9,620 forecasted values • Baseline scenario is the time to “prime” the models; includes adding in correlation • Each additional scenario is time to re-run model when a single distribution parameter is changed