Realism in Assessment of Effort Estimation Uncertainty :

1. Realism in Assessment of Effort Estimation Uncertainty: It Matters How You Ask By Magne Jorgensen IEEE Transactions on Software Engineering Vol. 30, No. 4, April 2004 Presented by Debra Dirlam Oct 20 2004

2. Effort Estimation Uncertainty • How sure are you of this estimate? • Managers depend of your estimate and your level of uncertainty about the estimate. • For sureness in manager’s decisions • For bidding on contracts • For project contingency buffers

3. It matters how you ask • How should you frame your request for uncertainty information about an estimate? • Obvious wrong way: “You don’t believe that it will take you more than 1700 hours, do you?”

4. Traditional Framing of the question • Estimators are asked to provide the minimum and maximum effort values based in given confidence levels • Confidence level usually 90% • “What is the minimum and maximum effort and be 90% sure?”

5. Alternative Framing of the question • Estimators are asked to assess the probability of the actual effort being higher or lower than a certain value. “How likely is it that the project will take more than 1700 hours?”

6. To Prove: • Alternate framing provides greater realism and more useful information • Traditional: Give me an estimate that you are 90% certain. • Alternate: Give me an estimate and tell me your certainty.

7. Research Steps Step 1 • Identify the size of the systematic overconfidence & understand the reasons • Found overconfidence high • When estimates claimed to be 90% confident they actually were only 60% on target • Level of overconfidence supported by other studies • Reasons… later

8. Research Step 2 • Looked at formal effort estimation uncertainty models designed to replace expert judgment • Some models could remove overconfidence at expense of widening the min-max interval • Conclude that current models could not replace expert judgment • More promising approach is to support expert judgment

9. Research Step 3 • Evaluated several strategies for judgment support in student experiments • One evaluated the framing variant and gave promising results • The experiment was replicated with software professionals

10. The Software Professionals Experiment • 29 experienced software developers & project managers • Paid to participate • Divided randomly into 2 groups • After giving estimate of most likely effort, half were asked “Tell me the interval in which you are 90% confident” And the other half were asked “Tell me the probability that the actual effort will be between 50% to 200% of your estimate”

11. The Software Professionals Experiment - Training • 10 real world software projects were estimated • Used expert judgment and an “experience database” of 5 similar projects • Feedback given after each estimate • Asked to reflect on performance

12. The Software Professionals Experiment – The Estimations • 30 software enhancement tasks previously conducted in a large telecom company • Estimate of 1st task was based on an “experience database” of 5 previously completed tasks • Estimate of 2nd task was based on the “experience database” and the feedback of the 1st task

13. Results • A hit rate similar to average confidence indicates good correspondence • Traditional framing shows slow approach to correspondence • Alternative framing shows a close to perfect correspondence on all sequence of tasks.

14. 4th Step – Full Scale Industrial Experiment • 2 medium sized Norwegian software development companies • No formal estimation process in place • All estimates based on expert judgment • Company projects and employees were similar • 18 months, projects >10 hours < 8 months • Projects were independent of each other

15. Industrial Experiment Design • During estimation phase, asked to complete questionnaire on effort estimation uncertainty assessment with either Traditional or Alternative framing. • Framing type was randomly chosen with 47 traditional framings and 23 alternative framings for a total of 70 projects • Possible for an estimator to have chance to do both framings • No feedback

16. Industrial Experiment Results • Results were similar to previous experiment • Traditional: 90% confidence corresponded to 74% hit rate • Correspondence better in the Alternative framing: 87% hit rate to 88% confidence • Analysis of any systematic favoritism: none or against Alternative framing

17. Discussion of Results • The 2 framing provide the same statistical problem • Looks how software professionals perceive and perform the uncertainty tasks in the 2 framings • Finds 2 important differences

18. Differences in How Software Professionals Perceive and Perform in the 2 Framings • Seems to be a better fit between Alternative framing and the format of historical estimation data. • The Traditional framing requires more complex analytical skill • Uncertainty estimates are highly intuitive

19. Differences in How Software Professionals Perceive and Perform in the 2 Framings • Software professionals may have goals other than realism in uncertainty estimates • Worry about providing meaningless wide intervals • Providing narrower intervals and more confidence evaluates to more skill as mistakenly perceived by managers • In Alternative framing the interval is not provided by the estimator and cannot be used in skill evaluation

20. Limitations • Cannot be generalize to all contexts • Evaluated only high confidence uncertainty assessments for the Traditional framing and wide ranges for Alternative framing. Results may be different for other values. • Not enough realism. Used questionnaires, importance of the role of requestor.

21. Conclusions • The best approach to assessing the uncertainty of effort estimates depends on many factors • Skill of estimators • Availability of information about previous projects • Type of information about the project to be estimated • Other factors

22. Conclusion - continued • The variety of factors does not lead to general laws to govern the assessment • The lack of general laws does not mean that all choices are equally good • The use of alternative framing is better supported by empirical evidence than the use of traditional framing. • Use Alternative framing