Schopenhauer's Proof For Software: Pessimistic Bias In the NOSTROMO Tool (U)

1. Schopenhauer's Proof For Software: Pessimistic Bias In the NOSTROMO Tool (U) Dan Strickland Dynetics Program Software Support daniel.strickland@dynetics.com • Ashley Mathis • THAAD Project Office System Software Engineering • ashley.mathis@mda.mil

2. Overview (U) • Background • Pessimism in Code Size • NOSTROMO Tool • NOSTROMO Demo

4. Background (U) • In 2003, TPO and Dynetics presented a methodology on using Monte Carlo simulation and uncertainty in COCOMO II to estimate risk in estimates • In 2004, TPO and Dynetics presented updated methodologies to include Auto-Generated Code and other updates to NOSTROMO • A new methodology for estimating pessimistic size growth has been incorporated into NOSTROMO

5. Boehm Horn Curve (U) • Size (and effort) should converge over time relative to development phase • During Concept of Operation, estimates can be 50% to 200% of the actual final size • Some types of software are always on the “bottom” part of the curve – always growing in size [Ref (1) - SOFTWARE COST ESTIMATION WITH COCOMO II (P 10) - BOEHM 2000]

6. Code Size Growth (U) • Code size estimates (and subsequent cost/schedule estimates) are required long before software requirements are baselined and finalized • Code estimates grow to any number of factors: • Requirements Volatility • Auto-Generated Code • Poor Estimating Techniques • Many large, complex programs exhibit constant code size growth outside the boundaries of the “horn curve” • We define pessimism as extending the boundaries of the expected code size growth to historical limits with only growth potential – “things can only get worse”

7. Three Historical Projects – Size Growth (U) • Three projects that have completed development or are very near completion • Graphs represent code size estimates over time for new code and subsequent Effective SLOC • Graphs show growth realization in new and ESLOC growth over time

8. Three Historical Projects – Percentage Size Growth (U) • Projects new code size grew 90-180% from original estimates • Projects ESLOC size grew 40-120% from original estimates • Programs exhibited extensive growth afterCritical Design Review (CDR) – during Coding and Unit Testing • Percentage growth is outside of the boundaries expected by the horn curve

9. Pessimism Curve (U) • Plot the data points of the three projects in the same manner as the relative size of the horn curve • Assume that the “ceiling” for Pessimism is the final size x and the “floor” encompasses the most extreme data points • Where data is unavailable, use the horn curve values and some extrapolation if necessary * ** * Value taken from horn curve – no like value in data ** Value extrapolated – average of previous and subsequent phases

10. Smoothed Pessimism Curve (U) • Mapping the points of the pessimism curve against the horn curve show an exponential curve of greater growth potential • Using MS_Excel, we added an exponential trendline to smooth the pessimism curve and identify a formula for use (R-squared value = 0.9666) • Using the smoothed curve, we can calculate the ranges for our pessimism curve as tied to development phase • We used this new range (inverted) as boundary conditions for size in NOSTROMO • NOSTROMO uses a triangle distribution between high and low boundaries for size

11. Original NOSTROMO Concept (U) Notional Obscurity STatistical Risk Observation MOdel DATA PAGE NEWT NOSTROMO REPORTS • NOSTROMO first takes normal COCOMO II inputs • NOSTROMO accounts for uncertainty in the settings • NOSTROMO uses Monte Carlo to simulate hundreds of COCOMO II estimates using the inputs and uncertainties • NEWT – NOSTROMO Entry Writing Tool – a Delphi polling tool that captures Uncertainty

12. New NOSTROMO Concept (U) PESSIMISM AGC CALCULATOR DATA PAGE REPORTS NEWT NOSTROMO DEFAULT UNCERTAINTY COMBINATION • NOSTROMO 0.3.15 offers: • multiple methods for inputting Uncertainty • combination of subcomponents • more reporting capability • AGC methodology • Pessimistic SLOC estimation

13. Distributions of Uncertainty (U) Level 1 - Certain NOSTROMO assumes ceiling and floor limits of the highest and lowest default values for each COCOMO II Scale Factor and Cost Driver Level 2 - Low Uncertainty - Normal Level 3 - Medium Uncertainty - Triangle Level 4 - High Uncertainty - Uniform Setting n Setting n+1 Setting n-1 NOSTROMO changes the distribution and sets the high and low points based on the level of uncertainty with each factor

14. Microsoft Excel 2002 worksheet with attached macros and Visual Basic code Currently on version 0.3.15 (as of 10/19/06) Does not use outside applications for Monte Carlo simulation (standard random number generation from Visual basic) Limited testing performed on a proof-of-concept application Suggest limiting Monte Carlo run size to 500-2000 runs Generates two pages of output – Data and Charts Data page contains all outputs from Monte Carlo simulation, charting data, and histograms Charts page contains Confidence Intervals for output of COCOMO II and Putnam models, charts, and histograms NOSTROMO – Data Sheet and CM Control Sheet are Developer pages and should not be removed NOSTROMO Application (U)

15. NOSTROMO Input Sheet – Notional Data (U) Software Item Name Scale Factors Function Buttons Code Size Pessimism Toggle Conditional Formatting Cost Drivers Pessimism toggle automatically sets the Requirements Evolution and Volatility to 0% to avoid double counting Iterations Putnam Productivity

16. NOSTROMO Pessimism Worksheet (U) Allows the user to develop their own relative size numbers for Pessimism and use in NOSTROMO

17. NOSTROMO Outputs – 5000 SLOC (U) 5000 SLOC – Pessimism On 5000 SLOC – Nominal REVL – No Pessimism Pessimism accounts for an 18% increase in Size in the 50th Percentile NOSTROMO Run

18. NOSTROMO Outputs – 500000 SLOC (U) 500 KSLOC – Pessimism On 500 KSLOC – Nominal REVL – No Pessimism Pessimism accounts for a 20% increase in Size in the 50th Percentile NOSTROMO Run

19. Demonstration

20. Future NOSTROMO Additions and Efforts (U) • Addition of Exception Conditions from Ray Madachy’s Expert COCOMO • Transition to an application environment (MS-Access or .NET) • Expansion of the NOSTROMO tool to address other COCOMO Family models with uncertainties • COSYSMO – especially relevant in development of complex DoD systems • COSOSIMO – System of Systems models are becoming increasingly prevalent in DoD efforts • Expansion of the NOSTROMO methodology and tool into Readiness Level models • Software Readiness Levels (SWRL) • Technology Program Management Model (TPMM)

21. Conclusion (U) • Software size in many development environments has a tendency to grow, even late into the project development • Software estimates need to address any pessimistic growth potentials and software estimators have an obligation to report the risks associated with developments that continue to grow in size • New NOSTROMO methodology incorporates pessimistic sizing data to predict the worst-case scenario Analysis of size growth potential gives valuable insight into the true risks of software development