Agenda

1. Joint Analysis of Cost and Schedule (JACS) Australian Department of Defence2nd Cost Estimation Conference 29 - 30 October 2012 Alfred Smith, CCEAJennifer Kirchhoffer, CCEA  Los Angeles  Washington, D.C.  Boston  Chantilly Huntsville Dayton Santa Barbara  Albuquerque  Colorado Springs Goddard Space Flight Center  Johnson Space Center Ogden Patuxent River  Washington Navy Yard Ft. Meade Ft. Monmouth Dahlgren Quantico Cleveland Montgomery Silver Spring  San Diego Tampa Tacoma  Aberdeen Oklahoma City  Eglin AFB  San Antonio New Orleans Denver Vandenberg AFB

2. Agenda • What is JACS? • Overview of the JACS modeling process • Key reports from a well constructed JACS model • Concluding remarks

3. What is Joint Analysis of Cost and Schedule (JACS)? • Cost, schedule and risk assessments traditionally have been performed by separate teams of professionals • In recent years, it has become more common for the cost analyst to report a “risk adjusted” result as a budget recommendation rather than a point estimate • However, it appears that cost uncertainty models routinely: • try to force a 70 or 80% cost result into the point estimate schedule • ignore risk management team statements like “High probability this event will occur and if it does, the consequence will be severe” Joint Analysis of Cost and Schedule is a disciplined, systematic and repeatable process to integrate three critical pieces of information: Cost, Schedule, Risk

4. Current Approach to Model Cost Estimating Uncertainty c Cost = a + bx Historical data points Cost • Sources of Uncertainty: • Cost estimating method • Cost method inputs • Focus is on estimating total cost uncertainty with limited influence from duration uncertainty or potential events that may influence cost/schedule. Input, e.g., weight

5. Evolving Trends in Uncertainty and Risk Analysis • Capture schedule uncertainty on Time-Dependent Costs • Inclusion of Discrete Risks (5x5’s) Time-Dependent (TD) [Level of Effort - LOE] Risk 1 Translate 5x5 into probability of occurrence times uncertain consequence which can impact cost and/or duration of one or more tasks Risk 2 2 1 . . . n Risk n

6. TD = Time-Dependent Cost, e.g. ‘marching army’ cost Integrated Risk & Uncertainty Landscape – the JACS Paradigm TI = Time-Independent Cost, e.g., Materials Uncert Uncert TI $ Uncert TI $ Uncert TD $ = Segment Duration X Burn Rate Uncert Uncert Project Start TI $ Uncertainty TI $ Duration Uncertainty Uncert Uncert Project End TI $ Uncert Probability of Occurrence Uncert Uncert TI $ Uncert Task Duration TI $ Risk Register TD $ = Segment Duration X Burn Rate Burn Rate Uncertainty Uncert

7. Agenda • What is JACS? • Overview of the JACS modeling process • Key reports from a well constructed JACS model • Concluding remarks

8. The JACS Process:Develop the Analysis Schedule Risk Sched CollectSched Data Create Analysis Schedule ValidateBeforeContinuing Cost

9. The Need for an Analysis Schedule • Joint analysis of cost and schedule begins with a model of the schedule logic • Serves as the backbone for the analysis • Cost, risks and uncertainty are mapped into the logic to assess impacts • Project/program integrated master schedules (IMS) are unsuitable for this role • They are generally too big, complex and too detailed • Logic common in an IMS can be a problem for a JACS analysis (e.g., constraints) • A JACS appropriate schedule must be created from available data (including the program IMS) • Typically referred to as an “analysis schedule”

10. Attributes of an Analysis Schedule • Captures the major work-flows of the project IMS • Provides insight into major cross-dependencies within or across management responsibility boundaries • Creates a solid framework to capture cost / schedule uncertainties and discrete risk events • Structured around management/ budget responsibility • Allows mapping of budgeted work effort to schedule scope • Aligns with cost/budget data • Identifies key tasks that support major deliverables/ tracking items • Detailed IMS step by step work items and task flows are combined while maintaining critical path logic • Has traceability and transparency to the more detailed IMS

11. The JACS Process:Map Costs to Schedule Tasks Risk Sched CollectSched Data Create Analysis Schedule Update Analysis Schedule Cost Collect Cost Data Identify as TD or TI Map to Sched Activities

12. Mapping of Cost to Schedule • Easier to map cost model results to a schedule model rather than replicating schedule network logic in a cost model • Schedule models are generally populated with throughputs, that is they don’t allow equations to estimate the cost of one task based on the cost of another or some technical characteristics • Mapping cost estimate to a schedule model is simplified by: • Unifying cost (often product based) and schedule (often task based) work breakdown structures • Specifying Time Dependent and Time Independent costs and their uncertainty separately • Defining how the TI or TD cost is phased over the task duration Total Cost TD Cost TI Cost TD Phasing TI Phasing

13. The JACS Process:Mapping the Risk Register to Schedule Tasks Risk Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Sched CollectSched Data Create Analysis Schedule Update Analysis Schedule Cost Collect Cost Data Identify as TD or TI Map to Sched Activities

14. Discrete Risk • Defined as: If risk event A occurs, there is a cost consequence or opportunity. The probability of A occurring is x% • Often modeled as a separate task inserted into the schedule network • If there are only a few such risk events, treat as discrete what-if cases (event cost or schedule impact is either “in” or “out” of the estimate) • Point estimate includes the full impact (when there are few) • If there are many such risk events, model using the Yes/No distribution (also known as the Bernoulli distribution) • When there are many, there is no standard on how to treat the Point Estimate: • Include none and assess separate from the Point Estimate? • Include all (worst case scenario)? • Include the sum of the expected values?  this is a common approach • The risk register should account for: • Uncertainty of the cost consequence or opportunity • Correlation across duration and cost uncertainties • Probabilistic branching  rarely attempted

15. Risk Management conventions* Consequence 1 Minimal or no impact 2 Additional resources < 5% 3 Additional resources = 5-7% 4 Additional resources = 7-10% 5 Additional resources > 10% Likelihood of Occurrence 1 Remote (10%) 2 Unlikely (30%) 3 Likely (50%) 4 Highly likely (70%) 5 Near certainty (90%) Opportunities Should have a separate matrix to address potential opportunities to save (not addressed in our example) 5x5 Matrix Definitions* Consequence *Note: Taken from Risk Management Guide for U.S. DoD Acquisition Every Agency will set its own standards for these values

16. The JACS Process:Assigning Uncertainties Risk Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Assess Event Cost and Duration Uncertainty Sched CollectSched Data Create Analysis Schedule Update Analysis Schedule AssessDuration Uncertainty Cost Collect Cost Data Identify as TD or TI Map to Sched Activities Assess Cost Uncertainty

17. The Only Certainty is Uncertainty • Most common method to address uncertainty is to assign distributions to uncertain elements and run a Monte Carlo simulation • Objective Uncertainty Distributions • Derived from historical data • Something you can defend mathematically and historically • Subjective Uncertainty Distributions • Based more on expert opinion than statistical analysis • Often necessary due to lack of information to characterize it objectively • Every duration, cost and consequence in the model is generally an estimate and therefore uncertain • Time Independent Cost Uncertainty • Time Dependent Cost (Burn Rate or Resource Utilization) Uncertainty • Duration Uncertainty • Discrete Risk Uncertainty –Probability of Occurrence • Uncertainty should be applied in a consistent manner across the entire model

18. Typical Uncertainty Distributions Note: Low/high are defined with an associated percentile (by default 15/85). Min/Max are the absolute lower/upper bound (also known as the 0/100). Some policies require truncation at zero.

19. The Double Counting Time Dependent (TD) Cost Uncertainty Dilemma • Consider how TD cost is calculated • Typically calculated using: Duration * Cost/Day (duration in days) • Cost/Day can be derived from similar, completed project totals • Cost/Day factor may already capture cost and duration uncertainty • Uncertainty on Duration and Cost/Day factor may be double counting • However, basis for Cost/Day must be carefully understood • Cost/Day factor may change as the duration changes • Shorter duration achieved by using more resources ($/day larger) • Shorter duration achieved by using more expensive resources ($/day larger) • Longer duration a consequence of scarce resources ($/day smaller) • In these contexts, uncertainty on Duration and Cost/Day is appropriate • Correlation between the two should be considered as well

20. The JACS Process:Apply Correlation, Validate then Run Risk Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Assess Event Cost and Duration Uncertainty Apply Correlation Sched CollectSched Data Create Analysis Schedule Update Analysis Schedule AssessDuration Uncertainty Validate File Run Analysis Cost Collect Cost Data Identify as TD or TI Map to Sched Activities Assess Cost Uncertainty

21. Apply Correlation • JACS models developed in any tool will have limited “functional” correlation between uncertain elements (correlation due to model mathematics) • Consider applying correlation to ensure elements that should, do move together • Tools should allow application of correlation across any uncertain elements • Just because you can apply correlation, does not mean you should! • Correlating Dur with TD may be double counting if TD is modeled as a function of Duration • Ideally, measure the correlation present in the model first, then apply as needed

22. Perform a Comprehensive Health Check to Ensure Model Validity • Prior to running an integrated simulation, the user should review the file to ensure that there are no potential issues within the file • There are many commercial tools that will perform a schedule health assessment • These tools look for violations of schedule model best practices (e.g., task without predecessor) • A JACS Health check is more comprehensive and should also uncover issues such as: • Critical issues: e.g., cost not phased, invalid uncertainty, duration with no cost, invalid correlation • Warnings: e.g., uncertainty on zero cost, risk event with zero probability, baseline outside uncertainty • Information: e.g., extraordinary float, risk event turned off, duration without uncertainty, no correlation

23. The JACS Process Risk Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Assess Event Cost and Duration Uncertainty Apply Correlation Sched CollectSched Data Create Analysis Schedule Update Analysis Schedule AssessDuration Uncertainty Validate File Run Analysis Cost Collect Cost Data Identify as TD or TI Map to Sched Activities Assess Cost Uncertainty

24. Agenda • What is JACS? • Overview of the JACS modeling process • Key reports from a well constructed JACS model • Concluding remarks

25. Typical Output from a Cost Only Uncertainty Analysis • Cost uncertainty is generally performed on total costs • Rarely linked to schedule uncertainties • Unable to relate a specific cost result to a specific schedule result • No insight into uncertainty by year, or the impact of schedule slips

26. A JACS Model Relates Uncertain Cost with Uncertain Duration Each dot is the cost and schedule result from one trial New View Shows Cost Aligned with Schedule Lower left identifies joint probability of meeting BOTH 70% cost and schedule (58.4%) 70% Cost Confidence Level (CCL) Indicates Reserves Capture Schedule Growth

27. Annual Funding Charts • The JACS Model provides total and annual uncertainty results • Identifies both WHAT the uncertainty is and WHEN it is Point Estimate vs Annual Uncertainty

28. Additional Views Are Possible Point Estimate similar to Mean Need to use Reserves to move point estimate towards the Mean

29. Additional Views Are Possible • There are many tools available and they all produce some version of most of these charts

30. Most Misunderstood Chart:Tornado • The term “Tornado” is used in a variety of contexts across schedule, cost and general uncertainty analysis; tool documentation and the literature • Various definitions include: • Find Cost Drivers: Create a low and high “what-if” case for every “cost driver” based on their 10/90% values (evaluated one at a time)…find driver that has most impact • Problem: ignores correlation effects; does not address drivers influenced by multiple distributions • Find Uncertainty Drivers: Measure the correlation between each defined input distribution and the output of interest. The highest correlation identifies find distribution that has the most impact on the total uncertainty. • Problem: Element that is highly correlated with output may have nothing to do with that output • Hybrid: Sort the trial results by cost driver (one at a time) to find associated bounds on the output mean or selected percentile • Problem: How you bin the trials has massive effect on results; may require huge number of trials to obtain stable results • Brute Force: Use one of the above methods to find the top 10, then run the simulation 10 times turning one at a time and record the impact on the output of interest. • Problem: Very tedious, time consuming and may be misleading if its loss triggers unexpected conditions in the simulation • Conclusion: Beware of the Tornado chart!

31. Tornado Chart Best Practices • Total Cost Drivers: (left chart) • Measure impact on inflated results, not constant year cost results • Run simulation to find combined uncertainty results for 10/90% • Total Uncertainty Drivers: (right chart) • Adjust the algorithm to account for applied correlations • Identify how many trials used (demonstrate sufficient) • Notice the answers are quite different 7k Trials Required

32. Prototype Integrated Time-Based View of Costs, Schedule, Budget, and Risks • Solid line is the point estimate (BCWS) • Dots are cost/schedule uncertainty results at various milestones • X- Axis identifies when key risk register events occur • Size of symbol indicates impact, color indicates probability (or type)

33. Agenda • What is JACS? • Overview of the JACS modeling process • Key reports from a well constructed JACS model • Concluding remarks

34. What Data Do We Need? • Schedule + Risks + Costs = JCL Model • Can use any of the below, as long as you have one data source for each category • Schedule • Detailed IMS • Simple schedule with just a few moving parts • Costs – preferably time phased • Budget data • Lower level cost data (LCC databases) / EVM data • Parametric costs • “Risks” • Risk management system • What –if’s • Basic uncertainty Keep it simple and use what you have

35. Why JACS is Becoming Popular • Delivers an integrated view to Project Managers: • Schedule probability of success • Cost probability of success • Impact of discrete program risks • Results of any number of what-if scenarios • Both total and annual funding reserve requirements • For NASA: Regulatory requirement (7120.5 E) • Identify a cost and schedule range by milestone KDP (~milestone) B • Baseline program to a specific joint probability level by KDP (~milestone) C

36. Questions?