Making Connections to Existing CERs with QUELCE*

1. Making Connections to Existing CERs with QUELCE* COCOMO Forum November 2015 Bob Ferguson *Quantifying Uncertainty in Early LifeCycleEstimation

2. Session Agenda • Overview of QUELCE • CERs1 in Estimation • Connecting the QUELCE BBN2 to CERs – Glue Nodes • Summary 1 Cost Estimating Relationships 2 Bayesian Belief Network

3. QUELCE • QUELCE is a method for analyzing the effects of changing assumptions and other risks inherent in product development. • Rather than focusing on the consequences of each risk, the method tries to analyze the cascading effects of multiple changes. • The current presentation describes ways to connect the QUELCE probability network (BBN) to many different cost estimation relationships.

4. QUELCE Intended to Inform CER Inputs (Example) Need to mapQUELCE change drivers (from the left) to the CER input parameters (on the right) • Acquisition Environment • A.1 Acquisition Category (ACAT) Status • A.2 Governance, Policies, and Standards • A.3 External Interdependencies / Coordination • A.4 External Stakeholders • A.5 External Events, Funding • A.6 Other: Acquisition Environment • A.7 Capability Definition • Acquisition Management • B.1 Acquisition Strategy • B.2 Contracting • B.3 Management Structure • B.4 Program Scope, Requirements • B.5 Budget • B.6 Schedule • B.7 Staffing • B.8 Facilities, Support Technology, and Equipment • B.9 Program Information Management • B.10 Program-Contractor Performance • B.11 Other: Acquisition Management • Technical Solution • C.1 Conceptual Design / Specification • C.2 System Architecture and Design • C.3 Production and Construction • C.4 Certification and Accreditation • C.5 Deployment, Operations, and Support • C.6 Technology Maturity / Readiness • C.7 Estimated Complexity / Difficulty • C.8 Supply Chain Products • C.9 Other: Engineering Solution / Work Products

5. Overview of QUELCE Method 1. Use QUELCE Repository to Populate Driver State Matrix 2. Evaluate Cause and Effect Relationships and Reduce Explosion via Dependency Structure Matrix 3. Develop BBN Model and Assign Conditional Probabilities to BBN Model 4. Calculate Cost Factor Distributions for Program Execution Scenarios 5. Monte Carlo Simulation to Compute Cost Distribution 4. Cost Factor Distributions by Scenario of Change DAES SARS SRDR 2. Dependency Structure Matrix QUELCE Change Driver Repository 5. Monte Carlo with Cost Estimation Tools 3. BBN Model 1. Driver State Matrix Complexity Reduction Modeling Uncertainty Legend:

6. CERs in Estimation • CAPE recommends four cost estimation methods: • Analogy: (early in program lifecycle -> MS A) • Parametric (Statistical) - CERs : (early in program lifecycle -> MS B) • Engineering (“bottoms up”): (MS C and later) • Actual Costs: (LRIP and later) • Different Types of Cost Estimation Relationships (CERs) • Elements of scope – things we build • Required process elements associated to scope • Required purchases • Scope • Estimate of volume, weight, size, capacity based on required deliverable characteristics. This may be LOC, Function Points, requirements, etc… • Product characteristics (quality attributes) such as speed of performance • Quantity of deliverable • Required Process Elements Associated to Scope • Certifications

7. Methods Employed in CERs • Scope Estimates • CER is a parametric model - an equation used to estimate a given cost element using an established relationship with one or more independent variables. • Analogy to past projects can be used to bound regression parameters. • Expert judgment recommendations based on experience and data. • Required Processes Associated to Scope • Built from the scope estimates by using internal history and a proportional formula. • Some of the required processes will be covered by a parametric tool. • Other Required Processes • Typically based on assigning a resource cost for duration of schedule or duration of development for a specific element of scope. • Purchases • Fixed cost based on count of need.

8. What Are Glue Nodes? • Definition • A glue node is a way to connect the output from the BBN in step 3 of the QUELCE method to the CERs for your estimation model. • Rationale • Glue nodes are used to incorporate the front end uncertainty and scenario analyses afforded by QUELCEin a practical manner by mapping the output of the BBN into existing parameters in the CER. • Example • The process elements of scope are uncertain based on need for innovation, product complexity, etc. Hence, we attempt to identify the related nodes in the model. Those nodes are connected to a glue node. • The probability distribution is calculated as the joint probability distribution of the parent nodes. • The glue nodes are mapped to selected parameters of the CER.

9. BBN Model with three Glue Nodes Product Challenge Scope (Size) Program Challenge

10. How to Use Glue Nodes • Each element of scope is likely to require different glue nodes • Schedule • The overall schedule will drive significant costs not associated to a specific scope, so different CERs are employed. • Schedule will be specific to a lifecycle. • Use • The BBN must be connected to the selected CERs via glue nodes. • The glue node embodies the joint probability of several other BBN nodes. • The glue nodes will be specific to the CER. • The BBN itself usually remains the same across different CERs. • BBN Analysis • BayesiaandAgenaRisk are example tools that support this function. • It is easy to add the needed joint probability distributions.

11. COCOMO II Example • We used 3 glue nodes • Size • Mike Ross code growth provides a useful size range • Median and a high percentile value are calculated • Product Challenge • We would like to think of product domain as providing a critical combination such as real-time-embedded very high reliability • Also novelty or invention adds to complexity • Program Challenge • Factors associated to contractor skill, program office flexibility, and number of contractors

12. COCOMO II Example - Glue Nodes • COCOMO parameter mapping -- • Select parameters for “Product Challenge” and “Program Challenge.” • Range is limited for each of the selected parameters. • Map BBN glue node values to COCOMO parameter values. • Some parameter scales had to be reverse ordered so that the effect had the appropriate direction BBN Outputs MappedCOCOMO value

13. Product Challenge Factors Italicized values are derived from the BBN; corresponding COCOMO values are directly below the BBN values.

14. Program Challenge Factors Italicized values are derived from the BBN; corresponding COCOMO values are directly below the BBN values.

15. Monte Carlo Modeling Each CER parameter is assigned a distribution based on the Glue node distribution BBN Outputs MappedCOCOMO value Monte Carlo simulation then enables a distribution of cost using the CER with input distributions

16. Summary Change Driver Scenarios modeled to inform CER inputs leading to cost distributions

17. Contact Information

18. BACKUP SLIDES

19. Change Drivers • Definition: A change driver is any assumption or decision that has the potential to change. • Decisions: • Selection of suppliers and contractors including test facilities • Selection of technology or component • Assumptions: • A technology, currently at TRL 6, will mature sufficiently by Milestone C. • A KPP goal (key performance parameter) can be achieved(KPP goals for software defined radios were not achieved). • Scope and feature definitions are fixed by Milestone B.

20. Change Drivers and Dependency Structure Matrix • “Change Drivers” • The specific assumptions and risks identified by program subject matter experts (SME) • SMEs are asked to make estimates of frequency of change • The Dependency Structure Matrix (DSM) • Relates changes by asking whether a second change event would be a consequence of an earlier one. • The matrix tends to be very large and must be reduced and reorganized into a triangular matrix in order to create the probability network. • SEI has an automated tool that makes this process much easier. It now takes about 30 minutes or less.

21. Modeling Schedule Impacts • QUELCE Addresses Schedule Slip • Use lifecycle synchronization points such as PDR and CDR. • Delays are common and usually have identifiable causes that have been modeled in the BBN. • The critical concern for schedule is whether specific change events occur that may cause schedule delays. • Synchronization Concerns • A critical dependency during the lifecycle is the need to synchronize development teams and tasks in order to do any form of product validation. Without this validation exercise, no forecast of delivery is trustworthy. • Schedule forecasts of deliverables are essential to the accuracy of the cost estimate.

22. Schedule Example: Causes of Schedule Delay • Supply Chain • Obsolete parts • Change of supplier • Technology Readiness and Manufacturing Readiness • e.g. GN (gallium nitride) is a potential replacement for Gas (gallium arsenide). Both electrical characteristics and manufacturing problems still need study. Software will interact with electrical characteristics. • Contractor – Program disputes • Approval of engineering change request • CDRL review delays • Limited availability of specific resources • Test equipment and facility • Funding Delays • … several others that are familiar to experienced program managers

23. Bayesian Belief Network • Directed Graph • A directed graph is constructed from the triangular matrix. • Each node gets a probability of occurrence and a probability of change effect from each parent node. • Probabilities are generated by SMEs and by data from past programs. • Stability of the BBN • Our expectation is the BBN should change rarely during most programs. • New risks or assumptions not tested would add a node. • As time passes, more and more nodes would be set to zero expectation of change. • BBN probabilities would consistently narrow over the course of time.