A BBN as it Would be Developed and Used in the QUELCE Method

1. A BBN as it Would be Developed and Used in the QUELCE Method Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 Robert W. Stoddard Jim McCurley 24 October 2013

2. Introduction • QUELCE (Quantifying Uncertainty for Early Lifecycle Cost Estimation) is a multi-year research project led by the Software Engineering Measurement and Analysis (SEMA) team within the SEI Software Solutions Division. • Research team membership comprises SEI technical staff with cost estimation background in collaboration with several external faculty (Dr. Ricardo Valerdi, Univ of Arizona, & Dr. Eduardo Miranda, CMU). • This research is motivated by (1) the WSARA Act requiring cost estimates pre-Milestone A and (2) DoD’s need for more accurate cost estimation methods that provide continuous monitoring of changing assumptions and constraints.

3. Overview of QUELCE 1. Use QUELCE Repository to Populate Driver State Matrix 2. Evaluate Cause and Effect Relationships and Reduce Explosion via Dependency Structure Matrix 3. Develop BBN Model and Assign Conditional Probabilities to BBN Model 4. Calculate Cost Factor Distributions for Program Execution Scenarios 5. Monte Carlo Simulation to Compute Cost Distribution 4. Cost Factor Distributions by Scenario of Change SARs SRDR 2. Dependency Structure Matrix QUELCE Change Repository Queries of Historical MDAP Experience and Context 5. Monte Carlo with Cost Estimation Tools 3. BBN Model 1. Driver State Matrix Complexity Reduction Modeling Uncertainty Legend:

4. What is a Bayesian Belief Network (BBN)? • A probabilistic model shown in a graphical form consisting of nodes and arrows • Nodes represent factors • Arrows between factors represent cause-effect relationships (ideally), or correlated relationships (minimally) • Factors may be set at “observed” levels representing observed “evidence” • BBNs use traditional conditional probability and Bayesian calculations to update all “unknown” factors based on the latest “evidence”

5. Burglar Alarm Example of a BBN - 1 The following example of a BBN was derived from an example model shown on the AgenaRisk tool vendor website. This example helps to concisely articulate the operation and use of a BBN. Reproduced from Jensen 1996, as seen at http://www.agenarisk.com/resources/example_models.shtml

6. Burglar Alarm Example of a BBN - 2 The probabilities of this baseline model reflect both historical data and expert belief. (Re-produced from Jensen 1996, as seen at http://www.agenarisk.com/resources/example_models.shtml)

7. Burglar Alarm Example of a BBN - 3 “Mr. Holmes is working at his office when he receives a telephone call from Watson who tells him that Holmes’ burglar alarm has gone off.” Convinced that a burglar has broken into his house (alarm sounds -> burglary), Holmes rushes into his car and heads for home.” (Re-produced from Jensen 1996, as seen at http://www.agenarisk.com/resources/example_models.shtml)

8. Burglar Alarm Example of a BBN - 4 “On his way he listens to the radio, and in the news it is reported that there has been a small earthquake in the area (radio report -> earthquake). Knowing that the earthquake has a tendency to turn the burglar alarm on (earthquake -> alarm sounds), he returns to his work leaving his neighbors the pleasures of the noise.” (Re-produced from Jensen 1996, as seen at http://www.agenarisk.com/resources/example_models.shtml)

9. Example QUELCE Bayesian Belief Network

10. Benefits of Using a BBN Model for QUELCE – 1 • Models the uncertainty of program change drivers and their relationships using probability distributions. • No longer use single point estimates • Instead, we use ranges and distributions reflecting uncertainty • Provides continuous measurement and ability to update and re-estimate based on changes in program execution. • Once created, simple to run scenarios as new “evidence” is observed • Readily used to update cost estimates based on changing program conditions • Translates the net effect of program change driver uncertainty to the input factors of cost estimation models. • We use Monte Carlo simulation to translate BBN output distributions into distributions for input parameters of CERs

11. Benefits of Using a BBN Model for QUELCE – 2 • Enables the use of both objective (hard) information and subjective (soft) information as evidence to update our forecast. • Evidence can be factual observation, e.g. something has happened • Evidence can be subjective in terms of a person’s anticipation of an event occurring • Enhances ability to conduct “what-if” analysis in context of change drivers. • A scenario in the BBN is a collection of one or more change drivers observed or anticipated to occur with a specified probability • For each scenario, the BBN then recalculates and produces new outcome node distributions

12. Benefits of Using a BBN Model for QUELCE – 3 • Enables analysis and forecast with incomplete information (e.g., no status available for some change drivers). • Traditional statistical analysis requires entries for all modeled factors, e.g. in a regression equation • BBNs can provide updated assessments of all unknown factors based on whatever factors are observed • Provides the ability to determine which change drivers are most influential on downstream change drivers or BBN outcome nodes (e.g., project complexity, product complexity). • Any factor may be selected for evaluating sensitivity to any and all other factors • For any factor, a sensitivity “tornado” chart may be created depicting in descending order all other factors influencing this factor

13. Benefits of Using a BBN Model for QUELCE – 4 • Mayexplain the most likely state of affairs of upstream change drivers based on current observations of downstream change drivers. • Akin to diagnosing likely causes of today’s observations • Commonly used in medical diagnosis • May be used to diagnose what other change drivers led to the current state of affairs of known change driver occurrence

14. Future Work • Need to standardize the output nodes of the QUELCE BBN • Need to provide a method to connect the BBN output nodes to the inputs of commonly used CERs • Need to collect data from retrospectives and ongoing program executions to validate the performance of the QUELCE BBN and method

15. Summary • The DSM matrix captures the experts’ change probabilities for one change driver affecting another change driver (all possible pairings). • The BBN models the probabilistic relationships so that different change scenarios including cascading change may be evaluated. • For each scenario, the BBN produces probability distributions for the output nodes which will then be used to assign probability distributions to the input factors of the cost estimating relationships/tools.

16. Contact Information