Software Estimation: What, Why & How

1. Software Estimation: What, Why & How Nupul Kukreja 19th October 2012

2. Based OnSoftware Estimation: Demystifying The Black Art Steve McConnell Microsoft Press.

3. Agenda • What is an “Estimate”? • Purpose of Estimation • In-class quiz to gauge & develop your estimation skills • What are “Good Estimates”? • Estimation and Cone of Uncertainty • Estimation Techniques

4. Agenda: VBSE 4+1 View 5a, 7b. Options, solution development & analysis Dependency Theory Utility Theory 3. SCS Value Propositions (Win Conditions) 2a. Results chains 2. Identify SCS 3b, 5a, 7b. Cost/schedule/ performance tradeoffs 4. SCS expectations management Theory-W:SCS Win-Win 3b, 7a. Solution Analysis 5a, 7b. Prototyping 5. SCS WinWin Negotiation 6, 7c. Refine, execute, monitor & control plans 1. Protagonist goals 3a. Solution Exploration ControlTheory DecisionTheory 7. Risk, opportunity, change management 5a. Investment analysis, Risk analysis 6a, 7c. State measurement, prediction correction; Milestone synchronization

5. What Is An ‘Estimate’? • Estimate: Prediction of duration or cost of project • Target: Statement of desirable business objective Ex.: • We need to have product ready by Christmas • We must limit cost of next release to $3 million owing to budget constraints. • Commitment: Promise to deliver defined functionality at specific level of quality by defined date • Commitment != Estimate (doesn’t have to be )

6. Why Do We Estimate? • To determine if project’s targets are realistic enough to control progress to meet them • And NOT to predict a project’s outcome • Estimate vs. Target Gap: • ≤ 20%  Easy to control feature set, schedule, team size towards realization • Else  Not possible to control project towards successful realization  Targets need to be better aligned with reality 

7. Estimates vs. Plans • Estimates form foundations for plans • Plans don’t have to be same as estimates • If (Targets – Estimates) >> 1 • Plans must account for high risk • Else • Plans can assume less risk • Planning considerations that partially depend on accurate estimates: • Creating a detailed schedule • Prioritizing functionality for delivery • Breaking project into iterations etc.

8. Estimates As Probability Statements • Single-point estimates “assume” 100% odds of success; NOT realistic • Usually a ‘Target’ in disguise  • Must factor in uncertainty  i.e. project success follows a probability distribution 100% Nominal Outcome Nominal Outcome Probability Schedule (or Cost of Effort) Schedule (or Cost of Effort) Common Assumption More Realistic

9. What Is A Good Estimate? “An estimate that provides a clear enough view of the project reality to allow the project leadership to make good decision about how to control the project to hit its targets” As quoted from Steve McConnell’s book.

10. How Good an Estimator Are YOU? • Estimate the questions (in the following slide) to the best of your ability • Take a few WAGs if you will  • Fill lower/upper bound so that there is a 90% chance of including the correct value • This is a quiz on estimation and NOT on Googling skills. Estimate without any electronic, human or supernatural help 

11. Estimation Quiz

12. Love to Gamble? Let’s say we gave you the following proposition: • You win $1000 if the actual answer is within your range • You win $1000 by spinning the wheel below: • What would you choose? Your estimate/range or spin-the-wheel? $0 10% 90% Win $1000

13. Estimation Quiz 10% 90%

14. Implications of Choice Desirable: Set range just right so as to be indifferent between gamble and your estimate i.e. 90% chance, not more and not less, that answer is within range 90% Confident  90% of the time 90% of the answers within range (i.e. you get 9 answers correct on 9 of 10 such quizzes)

15. Accuracy of Estimates • Is it better to overestimate or underestimate?

16. Accuracy of Estimates Effort, Cost, Schedule Nonlinear penalty due to planning errors, upstream defects, high-risk practices Linear penalty due to Parkinson’s Law ←Underestimation Overestimation→ < 100% 100% > 100% Target as a Percentage of Nominal Estimate Penalties of underestimation more severe than those for overestimation. If you can’t estimate with complete accuracy, it’s better to err on the side of overestimation – Steve McConnell

17. Value of Accurate Estimates • Improved status visibility – compare planned progress with actual progress • Higher quality – Schedule-stress-related problems avoided i.e. not patchy code etc. • Better coordination with non-software functions – i.e. business function such as testing, document writing, training, financial projections etc. • Better budgeting – enable better forecasting • Increased credibility for development team - by providing realistic/reliable estimates • Early risk information – correct initial mismatch between project goals and estimates • Increased predictability – ability to know cost, schedule and functionality in advance. Need to make commitments to customers, investors, suppliers, the marketplace and other stakeholders!

18. Estimation Errors @#$%^&*( • Four generic sources: • Inaccurate information about project being estimated • Inaccurate information about capabilities of organization implementing the project • Too much chaos in project (i.e. moving target) • Inaccuracies arising from estimation process itself • In simple terms: missing, inaccurate, incorrect understanding of features, costs, schedule, effort etc., owing to the above factors

19. Cone of Uncertainty 16x Error range! 4x 2x 1.5x 1.25x 1.0x 0.8x 0.67x 0.5x 0.25x Initial Concept Approved Product Definition Requirements Complete User Interface Design Complete Detailed Design Complete Software Complete The more refined the software’s definition the more accurate the estimate

20. “Front-Loaded” Cone of Uncertainty The cone narrows itself only if you make decisions that eliminate variability. Else it’d just blow up even more 4x 2x 1.5x 1.25x 1.0x Software Complete 0.8x Detailed Design Complete 0.67x User Interface Design Complete 0.5x Requirements Complete 0.25x Approved Product Definition Initial Concept Milestones usually ‘front-loaded’  Improved estimation accuracy for first 30% of project i.e. from ±4x to ±1.25x

21. Using the Cone of Uncertainty • If using a single point estimate: • Come up with estimate • Use multiplying factors from previous chart for relevant milestone to get range

22. Estimation Techniques

23. Count, Compute, Judge • Count First – If you can count something directly please do so  • If you can’t count the answer directly, count something else (i.e. correlated to the item you wish to estimate) and compute the answer (preferably by using calibration data) • Use judgment as a last resort

24. Fermi-lize Your Estimation Skills • Enrico Fermi – Won a Nobel Prize in Physics in 1938. Well known for his creative and intuitive, even casual sounding estimates • A "Fermi question" is a question which seeks a fast, rough estimate of quantity which is either difficult or impossible to measure directly • In-class: Estimate the number of words in the today’s newspaper (Daily Trojan @ USC)* *DEN Students: You may use whatever newspaper you have access to at home or work. If not, you could just imagine one 

25. Fermi Decomposition • Figure out something that is known about the quantity in question • Estimate other things that may have a bearing on that quantity – it’s okay to have rough approximations • Sometimes you can just Google some numbers to extrapolate from there! • It’s ALWAYS possible to estimate and get ballpark idea about the quantity in question  • In-class 2: How many racket-ball or golf balls can fit in this class room

26. Individual Expert Judgment • Most common estimation approach • Experts = those are doing the task  • These are ‘task-level’ estimates i.e. specific tasks like feature development, testing etc., • Task level estimation: Decompose estimates into tasks requiring no more than 2 days of effort (rule of thumb to avoid estimation error)

27. Individual Expert Judgment Example of developer single-point estimates (not preferable) Example of individual estimation using best case and worst case. Provides better estimates and forces thinking of worst case estimates – leading to better overall range

28. Individual Expert Judgment Even Better: Compute a 3-point estimate including the most-likely case Compute expected case using the PERT formula: Expected Case = [BestCase + (4* MostLikelyCase) + WorstCase]/6

29. Individual Expert Judgment Compare estimates to actuals to improve estimation accuracy over time and ‘narrow’ the cone of uncertainty Magnitude of Relative Error: |(Actual – Estimate)/Actual|

30. Decomposition & Recomposition • Process: • Separate and estimate into multiple pieces • Estimate each piece individually • Recombine individual estimates into an overall aggregate estimate • AKA “bottom up” estimation or “Work Breakdown Structure (WBS) • Very important and highly used technique • Leads to quite accurate estimates

31. Work Breakdown Structure (WBS) • Example: Cost of owing and operating a car • Buy the car • Pay down payment • Pay taxes, licensing and registration fees • Insurance the car • Pay monthly loan installments • Operate and maintain the car • Pay semi-annual insurance payments • Fill car with gas when needed • Change oil every 3000 miles • Take car to oil change shop • Let them do work • Pay fees and taxes • Drive back • Other routine maintenance • Sell the car Estimate each piece individually and aggregate the estimates all the way to the top Law of Large Numbers: Overestimating some pieces will help cancel out some of the underestimates of the rest. Leading to better estimates

32. Estimation by Analogy • Create estimates of new project by comparing it to a similar past project • Can help create accurate estimates (by following the process below, instead of relying on memory) • Get detailed size, effort and cost results for similar previous project (WBS is preferable if possible) • Compare size of new, piece-by-piece to previous • Build up estimate for new project’s size as a percentage of old project’s size • Create an effort estimate based on size of new project compared to that of previous one • Check for consistent assumptions across the two

33. Proxy-Based Estimates • Very difficult to estimate SLOC count looking at a feature • Or expected #defects, #test-cases, #classes etc., • Proxy-based estimation: • Identify a proxy that is correlated with quantity to be estimated • Proxy is usually easier to estimate/count or available sooner in project • Compute estimate based on proxy and past historical data • Useful for creating whole-project or whole-iteration estimates but NOT for detailed task-by-task or feature-by-feature

34. Story Points • Unit-less measure of ‘complexity’ or ‘size’ of feature • Scales: • Powers of 2: 1, 2, 4, 8, 16 … • Fibonacci: 1, 2, 3, 5, 8, 13 … • #Story-Points per iteration = Velocity i.e. looking at past velocity estimate completion time of project (use historical data if available or forecast or run one iteration)

35. T-Shirt Sizing • Break features into various (albeit fuzzy) T-shirt sizes • Small, Medium, Large, X-Large, XX-Large etc. • Estimate Business Value and Ease of Realization using T-shirt sizes for each feature • Gauge overall feature value based on responses • Provides crude, sufficiently accurate, initial high-level estimates in the wide part of the cone

36. T-Shirt Sizing Score Chart • Can create one that suits the organization or use the one below as per Steve McConnell: • Use scores from above chart to compute approximate net business value for each feature (in sorted order)

37. Expert Judgments in Groups • Similar to planning poker  • Have each team member estimate pieces of project individually • Meet to compare the estimates • Reach mutual consensus as group • Without averaging the estimates. You may average the estimates but you still need to discuss individual results • Extremely effective technique to help improve estimation accuracy

38. Estimation by Tools • Helps perform tasks that can’t be done manually • Simulating project outcomes (i.e. sensitivity analysis) • Probability Analysis i.e. viewing the (cumulative) probability distribution of the estimates • What-if analyses • Serves as referee for unrealistic project expectations • Estimation of less common software issues • Works best if you have historical data for calibration

39. List of Available Tools • COCOMO II • Construx Estimate • Costar (commercial implementation of COCOMO II) • Price-S • SEER • SLIM-Estimate and Estimate Express • Your own home grown one?

40. Using Multiple Approaches • No single technique is perfect • Best to augment estimation with multiple approaches • Each approach works best in specific context • Convergence amongst estimates suggests a good estimate • Divergence helps see if something was overlooked or needs to be understood better

41. Special Issues in Estimating Size • Difficult to estimate raw SLOC • Employ various techniques for ‘estimating’ • Function Point • GUI Elements (i.e. counting and computing) • Dutch Method • Use-case points • Class points? (i.e. estimating #classes etc.) • COTIPMO  • Application Points • Object Points (not OOAD objects)

42. Special Issues in Estimating Effort • Quite difficult to get right • Various techniques maybe employed: • Informal comparison to past projects • Estimation software tools • Industry-average effort graphs • ISBSG Method (International Software Benchmarking Standards Group) • Has developed various equations fitting various types of projects that can be used for estimating effort (based on function-points for size)

43. Special Issues in Estimating Schedule • As evident, equally hard as the other two • Various techniques: • Schedule Equation (Courtesy Dr. Boehm ): • ScheduleInMonths = 3.0 x StaffMonths(1/3) • Informal comparison to past projects • Estimation software tools

44. Conclusion • Estimation is NOT an art (somewhat but not entirely) • Can effectively be executed as a science • Need not rely purely on intuition or memory • Improves over time especially if historical data is captured • Requires basic arithmetic to understand • Complex models can be created with the help of statistics – premise of most ‘tools’ • VERYYYYYYYYYYYYYYYY Critical skill-set to have in the 21st Century 