Marc Giombetti (TUM & MIT) Dr. Ricardo Valerdi (MIT) Dr. Stefan Wagner (TUM)

1. The influence of software quality requirements on the suitability of software cost estimation methods24th International Forum on COCOMO and Systems/Software Cost Modeling Marc Giombetti (TUM & MIT)Dr. Ricardo Valerdi (MIT)Dr. Stefan Wagner (TUM)

2. Outline • Motivation • Advantages and limitations of different software cost estimation methods • Understanding & estimating software quality • A controlled experiment – COCOMO II vs. Expert judgment • Conclusions & Future work

3. Formal estimation models or expert judgment ? (1/2) • Formal estimation models or expert judgment […] figure out when, and under what conditions, each method would be best.  This research adjusts this question and asks: When are formal models more useful than expert judgment; given a set of specific quality requirements. M. Jørgensen and B. Boehm, “Software Development EffortEstimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19.  

4. Formal estimation models or expert judgment ? (2/2) Formal Estimation Models • COCOMO, FP based models • Mechanical quantification step • Dependent on calibration data • Objective & repeatable ? • Industry usage? – Trust in a method which is not fully understood by PM? • Expert judgment • Checklist-based estimation, WBS, “gut feelings”, • Judgment-based quantification step • Dependent on individual estimator • Biased towards optimism • Only 15% of the journal papers analyzed judgment-based effort estimation* The difference between formal estimation models and expert judgment is not always clear  Combination of methods * M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53.  

5. Software Quality • Software quality is an important factor for the success of software of any kind. • How much software quality assurance is enough?* • For several industries quality is not negociatable! • One of the challenges of Software Quality is that"everyone feels they understand it“** • Quality requirements need to be taken into consideration while estimateing the cost of future projects. * L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95.   ** P.B. Crosby, Quality Is Free, Signet, 1980.  

6. ISO/IEC 9126 - Software Quality Model International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001.

8. The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (1/4) • Estimation for development time (no schedule compression) • Expert judgment : Top-down or bottom-up approach H1 H2

9. The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (2/4) • System requirements and project settings • Domain: Financial application • Type: Web application • Development team size: 1 Project Manager1 Senior developer,1 QA, 3 Junior developers • Expert estimation: Graduate students • Security: 5 students • Usability: 6 students

10. The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (3/4) • Research hypotheses • H10: In the case of additionalsecurity requirements There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate. • H20: In the case of additionalusability requirements: There is no significant difference between the average development time estimated using expert estimationand the intermediate COCOMO II model estimate. H1 H2

11. The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (4/4) • Experiment Goals • G.1. Analyze the relative impact of securityandusability requirements on the development time of a web-application using intermediate COCOMO II. • G.2. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by security requirements made using expert estimation compared to intermediate COCOMO II. • G.3. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by usability requirements made using expert estimation compared to intermediate COCOMO II.

12. Experiment: COCOMO II Estimate

13. Different mental models for the concepts security and usability??? Experiment: Results (1/2) H1 H2

14. Experiment Results (2/2) • T-Test & Wilcoxon signed rank test at 5% significance level: • Reject H1 There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the security enhancement. • Reject H2  There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the usability enhancement. • Threats to validity • Students in the role of expert estimators* • Classroom exercise: The requirements provided to the students were simplified, but chosen to be comparable to real requirements (in size & complexity). • Selection of quality requirements • Small dataset H1 H2 * P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176.

15. Conclusions & Future work • Perception of quality attributes influences the estimators, but not the COCOMO model estimate. • Human estimators underestimated security; overestimated usability • Influence of quality for cost estimation - Improvements • Conduction of experiment with more participants • Planned: Graduate students at TU Munich • Preferable: Software professionals with real estimation experience • Focus on other quality aspects (maintainability, reliability,…) • Conduction of a second experiment on the influence of usability (Focus on FP counting & COCOMO II)

16. Future work: COCOMO II – Estimating the influence of usability

17. Questions?

18. References • M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19.   • M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53. • L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95. • P.B. Crosby, Quality Is Free, Signet, 1980.     • International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001. • P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176.

19. Backup Slides

20. Expert estimation approach chosen by participants