INFO 631 Prof. Glenn Booker

INFO 631Prof. Glenn Booker Week 8 – Chapters 21-23 INFO631 Week 8

Basic Estimation Concepts Ch. 21 Slides adapted from Steve Tockey – Return on Software INFO631 Week 8

Estimation • Business decisions are always about future actions • Cash-flow streams for proposal are estimates • Correctness of business decision depends on accuracy of estimates • Estimation Topics • Basic Estimation techniques (Ch 21) • What they are and why made • General Estimation Techniques (Ch 22) • Estimation techniques and why you should use more than one • Allowing for Inaccuracy in Estimates (Ch 23) • Methods to address inaccuracy INFO631 Week 8

Basic Estimation ConceptsOutline – Ch 21 • What is an estimate? • Estimates and uncertainty • Cone of uncertainty INFO631 Week 8

What is an Estimate? • An estimate is... • Basically a quantitative prediction • Better question might be “why estimate to begin with? “an assessment of the likely quantitative result. Usually applied to project costs and durations and should always include some indication of accuracy (e.g., +/- x percent)” Reference: [PMI00] INFO631 Week 8

Why Use Estimation? • Business decisions look into future • Decisions based on factors whose value we might not be know yet • Or can’t be know for certain at the time of the decision • Example • How much does it cost to develop a product? • Won’t know for sure unless product is developed. • Goal: • Be close enough to allow you to make the right decision INFO631 Week 8

Estimates and Uncertainty • Estimates imply probability • Probability function • Set of all outcomes • Not symmetrical • General shape: Nominal Schedule (50/50 schedule) INFO631 Week 8

Estimates and Uncertainty • Estimate probabilities are not evenly distributed about a mean—there is a shortest possible schedule • Software projects • Usually have a a lower limit that can not be beat • No definite upper limit. • Almost always take a little more time and money • Broad range of possible outcomes • Called Variance (or uncertainty) • What happens when you arbitrarily adjust an estimate downwards? INFO631 Week 8

Estimates and Uncertainty High Uncertainty Low Uncertainty INFO631 Week 8

Sources of Estimation Uncertainty 10X • Will the customer want Feature X? • Will the customer want the cheap or expensive version of Feature X? • If you implement the cheap version of Feature X, will the customer later want the expensive version after all? • How will Feature X be designed? • How long will it take to debug and correct mistakes made in implementing Feature X? • Note: DeMarco and Lister coding wars found a 10x variation of difficulty of implementing designs of the same thing. 10X 10X 10X 10X INFO631 Week 8

Exact vs. At-or-less Estimates • Most decisions don’t need estimates of exact outcomes • Important thing is if actual outcome is at or below estimate • “Did project finish in exactly 10 months?” vs. “Did project finish in 10 months or less?” • Probability of being at or below any estimate is probability of hitting that exact outcome plus sum of probabilities of all outcomes less than it • Cumulative probability function—the integral of the probability function • Note: Impossible zone still applies • Question: Is less always better or worse? INFO631 Week 8

Cumulative probability function • Shape of cumulative probability function is very different than shape of the probability distributions • Since driving 500 miles in exactly 2 hours has a probability of zero then the probability of getting there in 2 hours or less is also zero. • Probability of getting at or less than the smallest outcome with a non-zero probability is the same as the probability of hitting exactly that same outcome. • Probability of hitting the second smallest possible outcome or less equals the probability of hitting exactly the smallest plus the probability of hitting exactly the next-to-smallest. • The probability of hitting any given outcome or less is the sum of the probabilities of hitting exactly the outcomes from that one all the way back to the beginning (e.g., add up the probabilities of all the arrive-exactly-on times from the given time back to the earliest possible arrival time). INFO631 Week 8

Cumulative Probability Functions INFO631 Week 8

The 50/50 Estimate • Target during estimation • Estimate equally likely to be above or below • 50/50 Estimate, Probability = .5 • Good target to measure estimation INFO631 Week 8

The 50/50 Estimate • From an estimators perspective • Target for 50/50 point • Estimate equally likely to be over or under • Test of estimators ability • See whether the actual outcomes are more than their estimates as often as they are less. • Consistent over/under means there is a systemic bias in the estimator's estimates INFO631 Week 8

Quantifying Estimate Uncertainty • Degree of uncertainty in estimate depends on how much is known • Further into a project you have a better understanding of what is going on • Early estimates have more uncertainty that later of the same thing • Cone of Uncertainty - quantifying and showing how it decreases over time • Best to understand by an example INFO631 Week 8

Quantifying Estimate Uncertainty • Example – develop a Cone of Uncertainty Project: Framus Estimate Act/Est Scope defined 7 mos 1.43 Requirements complete 11 mos 0.91 Design complete 10 mos 1.00 Code complete 9 mos 1.11 Test complete 9.5 mos 1.05 Project complete 10 mos 1.00 INFO631 Week 8

Relative Error in Estimates Project schedule 1.6x 1.25x 1.15x 1.1x 1.0x 0.9x 0.85x 0.8x 0.6x Scopedefined Requirements complete Design complete Code complete Test complete Project complete INFO631 Week 8

Relative Error in Estimates INFO631 Week 8

Cone of Uncertainty INFO631 Week 8

So Why Bother with “Cone” • Knowing “Cone” you can now adjust based on organizational history INFO631 Week 8

So Why Bother with “Cone” • Knowing “Cone” you can now adjust based on organizational history Lower Mean Upper Scope defined 0.40 1.16 1.91 Requirements complete 0.63 1.10 1.56 Design complete 0.73 1.05 1.36 Code complete 0.83 1.00 1.18 Test complete 0.95 1.01 1.07 Project complete 1.0 1.00 1.0 INFO631 Week 8

Putting Your Cone of Uncertainty to Work Graph of 50/50 estimates and uncertainty rages INFO631 Week 8

“Good” Estimates ‘Good” = allows decision make to make the right decision ‘Good” Estimates, actual outcome within the uncertainty range INFO631 Week 8

Expressing Estimate Uncertainty • Techniques to express • Plus-or-minus qualifiers • Ranges • Risk quantification • Cases • Coarse dates and time periods • Confidence factors INFO631 Week 8

Uncertainty and Ethics • The IEEE-CS/ACM Software Engineering Code of Ethics and Professional Practices [IEEEACM99] requires software professionals to quote uncertainties: 3.09. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work and provide an uncertainty assessment of these estimates. INFO631 Week 8

Cone of Uncertainty in Light of Fixed Schedule Over whole project uncertain features will decrease INFO631 Week 8

Key Points • Estimate is “an assessment of the likely quantitative result” • Estimates are needed to support decisions of factors not known until after the decision is made • Estimates don't have to be perfect • Estimates are inherently uncertain • That uncertainty can be expressed in a number of different formats • Your estimates should target the 50/50 point • Cone of Uncertainty shows how uncertainty changes over time • Measures your ability to commit INFO631 Week 8

General Estimation Techniques Ch. 22 Slides adapted from Steve Tockey – Return on Software INFO631 Week 8

General Estimation TechniquesOutline • Expert judgment estimation • Estimation by analogy • Bottom-up estimation • Statistical estimation • Estimation by multiple methods • Make assumptions explicit INFO631 Week 8

Expert Judgment Estimation • Ask one or more “qualified experts” • Software professional carrying out the decision-making study will probably be that expert in many cases • Advantages • Usable even when other techniques aren’t • Useful as a “second opinion” • Review other estimates • Simplest • Disadvantages • Most prone to bias and error • Most inaccurate • Depends on having credible “experts” INFO631 Week 8

Expert Judgment Estimation • May want to review expert judgment estimates • Important confidence-building step for all estimation techniques but particularly valuable here • If decisions are important enough, converge estimates from several experts • Some organizations choose two-thirds of the way between the lowest and the highest individual estimates • Nominal group estimate = lowest estimate + ( highest estimate - lowest estimate )*.6667 • Wideband Delphi is another possibility • Choose team, kick-off, individual estimate based on WBS, review estimates with team, merge together INFO631 Week 8

Estimation by Analogy • Assume thing being estimated is a lot like something we already know about • Base estimate of new thing on experience with old thing with allowances for differences • Usually more accurate then expert judging • Of course we’re assuming that there aren’t any other major differences that would influence the cost. • If there were, you’d need to be sure to account for them too. INFO631 Week 8

Estimation by Analogy • Process • Understand the new thing • Get detailed results of one or more analogies • List differences • Estimate impact of each difference • Build up estimate for new thing based on actual results of analogy project plus the differences INFO631 Week 8

Estimation by Analogy • Advantages • Usually more accurate than expert judgment • Relatively quick and easy • Disadvantages • Requires one or more analogy projects • Also requires good record keeping • If not analogous project • Might not be a good choice INFO631 Week 8

Bottom-up Estimation • A popular approach • Aka “module build-up”, “by-engineering procedures”, “top-down”, “decomposition”, ... • Developing cash flow streams using a Work Breakdown Structure (WBS) is an application of this estimation technique • Based on a statistical property called The Law of Large Numbers • Sum of errors in low-level estimates tends to be less than the error in a single estimate of the whole • In other words, lots of little errors tend to cancel each other INFO631 Week 8

Bottom-up Estimation • Process • Break whole into smaller parts • Finer decompositions usually better • Balance with effort to decompose and estimate parts • Estimate parts • Use appropriate techniques • Some will be overestimated and others underestimated • Sum estimates of parts • Overestimates tend to cancel underestimates • If bottom-level estimates don’t address system-wide factors, those will need to be accounted for INFO631 Week 8

Bottom-up Estimation • Advantage • Law of large numbers tends give more accurate estimates than single overall estimate • Disadvantages • Need detailed WBS • May not be practical early in a project • Systemic biases in low level estimates will be compounded • Easy to overlook project- or system-wide factors • E.g., planning, project management, documentation, testing, rework in a software project • Effort to create and roll up the detailed estimates may make other methods more attractive INFO631 Week 8

Statistical Estimation • Using an equation relating something countable at decision-time to the factor the decision is based on • Often called “parametric estimation” • Equation typically derived from historical data, i.e., completed software projects • CoCoMo-II, Price-S, Putnam, SEER-SEM, SLIM INFO631 Week 8

Model User’s Perspective: Cocomo II KSLOC = Source lines of code (thousands) B = Effort growth factor EM = Effort multiplier B = Effort growth factor Note: Based on a regression formula INFO631 Week 8

Model Developer’s Perspective • Process: • Identify observable factor(s) that drive the estimate • Gather as many valid data points as possible • Each point needs quantified factors and actual result • Be certain data is complete and consistent • Derive functional relationship between factor(s) and observed results • Optionally, verify that equation is a good predictor • Analyze its ability to match the historical data • This step can be complex and involve heavy-duty statistical analysis INFO631 Week 8

Statistical Estimation • Advantages • Can be the most accurate • Usually quick and easy to use once the functional relationship has been derived • Can be as sophisticated as needed • Can be self-tuning • Disadvantages • Require an adequate base of historical data • Significant effort along with mathematical and statistical knowledge may be required to derive and verify the functional relationship • Final equation(s) will be sensitive to the quantity, accuracy (consistent accounting practices across all of the historical data), and relevance of the historical data • Users may put too much faith in a model that hasn’t been validated INFO631 Week 8

Estimation by Multiple Methods • Sophisticated organizations use several estimation methods throughout the project • “Sanity check” other estimates • Preferred method often changes over the life of the project • Different methods typically perform better in the different phases of a project INFO631 Week 8

Preferred Estimation Methods by Project Stage Very Early (pre-requirements) Early (to end of requirements) Mid (to design complete) Late (to code complete) Yes, feeds into detailed, low-level estimates Yes, feeds into detailed, low-level estimates Expert Judgment May be only choice Reasonable as a second opinion Yes, feeds into detailed, low-level estimates Yes, feeds into detailed, low-level estimates Preferred, if possible Likely still preferred Analogy Low level, defect-rework estimates, reliability models Preferred(function points, subsystems, etc.) Yes, feeds into detailed, low-level estimates Statistical methods Maybe Maybe, based on requirements counts Decom-position Too early (no design detail) Preferred Preferred INFO631 Week 8

Make Assumptions Explicit • Almost impossible to estimate without making assumptions • Those assumptions can have a significant effect on the estimate • OK to say, for example “I estimate that the Framus project will cost between $0.8 and $1 million and take between 7 and 10 months” • Much better to say “I estimate that the Framus project will cost between $0.8 and $1 million and take between 7 and 10 months, but this assumes that the control algorithms aren’t more complex than expected and that the project is staffed according to Ron’s projections” • Making assumptions explicit allows consumers to decide whether or not they agree with the assumptions INFO631 Week 8

Make Assumptions Explicit • Notice that at no point did we try to present an estimate in the form: • “I estimate that the Framus project will cost $926,364.56 and take 9 months and 8 days.” • But that’s what people want to hear! • Something very precise, reassuring that we know exactly what will happen. INFO631 Week 8

Key Points • Expert judgment estimation uses a credible expert • Estimation by analogy creates an estimate from the actual outcome from one or more known, past situations together with an accounting of the differences • Bottom-up estimation divides the thing being estimated into smaller parts and builds up the estimate of that whole from the sum of the estimates of the parts • Based on “the law of large numbers” • Statistical estimation uses equations derived from relevant data • Convergence or divergence among the estimates can indicate how much trust should be put into those estimates • Preferred estimation technique(s) will probably change over the life of a software project • Assumptions should be made explicit INFO631 Week 8

Allowing for Inaccuracy in Estimates Ch. 23 INFO631 Week 8

Allowing for InaccuracyOutline • Knowledge drives accuracy • Allowing for inaccuracy • More effective ways to allow for inaccuracy: • Use ranges of estimates • Sensitivity analysis • Delay final decisions INFO631 Week 8

Knowledge Drives Estimation Accuracy • The more you know, the more accurate your estimates will be • Hint: • Historical data is easy to collect and incredibly valuable on later projects… INFO631 Week 8

INFO 631 Prof. Glenn Booker