Agile Software Development and ISO/IEC Software Quality Standards: Measuring Economic Benefits and Calculating Quantitat

1. Agile Software Development and ISO/IEC Software Quality Standards: Measuring Economic Benefits and Calculating Quantitative Yields Anca-Juliana Stoica, Margus Nael Royal Institute of Technology, KTH-ITC, Sweden 25th International Forum on Systems, Software, and COCOMO Cost Modeling University of Southern California, Los Angeles, CA November 2-5, 2010

2. Outline • Research Background • Agility vs. Discipline in Software Development • ISO/IEC Software Quality Standards • Research Objectives • Research Design • Case Studies • Techniques and Features Used • Framework for assessing a project • Metrics and Characteristics • Quality Metrics • Integrated Models for Measuring Economic Benefits and Calculating Quantitative Yields • Results • Conclusions

3. The Spectrum of Software Engineering Methods

4. Agile methods: Increased customer satisfaction Lower defect rates Faster development times Solution to rapidly changing requirements Examples: XP(Extreme Programming), Cristal Methods, ASD(Adaptive Software Development), DSDM, FDD, Scrum Plan-driven methods: Predictability Stability High assurance Examples: Cleanroom, PSP, TSP, CMM methods (Capability Maturity Models: Software CMM, Integrated CMM(CMMI)), ISO/IEC Software Quality Standards (9001 and the recent SQuaRE standards) Examples of Software Engineering Methods

5. Agility vs Discipline

6. Agility vs. Discipline

7. Challenge Balance the two approaches to take advantage of their strengths in a given situation while compensating for their weaknesses

8. Two suggested solutions • Software risk management approach • Software quality management approach, for structuring projects to incorporate both agile and plan-driven features in proportion to project’s needs.

9. Trends for the 21st Century Software Systems • For large – to very large scale software intensive systems, the Emerging Scalable Spiral process model • For small – to medium scale software systems, hybrid methods, using some other concepts in addition to software risk management, like: software quality management. Our focus here is on small – to medium scale software systems that need to balance agility and quality assurance.

10. ISO/IEC SQuaRE Series • Software product Quality Requirements and Evaluation • SQuaRE series of International Standards is based on 2 predecessors: • ISO/IEC 9126 (Software product quality) • ISO/IEC 14598 (Software product evaluation) • New general reference model consisting of 5 quality divisions: • Quality Management Division • Quality Model Division • Quality Measurement Division • Quality Requirements Division • Quality Evaluation Division.

11. Organisation of SQuaRE series of standards

12. SQuaRE Common Models • SQuaRE general reference model - navigation guide as a function of user’s task(s) • Software product LC model – views of internal, external software quality and software quality in use during software LC • Quality model structure –categorisation of software quality attributes into characteristics, sub-characteristics, and quality attributes. Remarks • Measuring and evaluating characteristics will assure quality • Organization must support the project with necessary equipment and knowledge • Create Quality Evaluation Project Plan

13. ISO/IEC SQuaRESoftware Product Quality Views

14. Research Objectives • Study if there are any conflicts between the application of agility principles and the ISO/IEC SQuaRE series of software quality standards • Guidelines on how to apply SQuaRE to agile (hybrid) software development • Real-life case studies for proving major benefits: • Software quality • Economic benefits.

15. Research Design • Data collection • Software project assessment framework based on agile/plan-driven characteristics • Experience base • Guidelines for applying ISO/IEC SQuaRE • Principles • How they apply to a specific project/development environment • Use of inductive approach • Real-life case studies with/without using SQuaRE • Software quality improvements • Economic benefits and other quantitative yields • Data analysis

16. Real-Life Case Studies Focus on two separate but similar real-life software development projects: • Project A -project without ISO/IEC SQuaRE • Project B -project with ISO/IEC SQuaRE Applied comparative analysis of projects and software metrics

17. Real-Life Case Studies Projects’ characteristics: • Modern Web-based small- sized software business systems • Stakeholder committment based on Theory W principles • Incremental development and delivery • Balancing agility and discipline • Software quality management done through the whole life-cycle • Use of a tailored software engineering method.

18. ProjectA/Project B • To publish fresh publications, e.g. news articles from news providers, via WAP or GPRS to the customers of a tecommunication company • Publications are delivered to application by RSS and server creates pages that are suitable for mobile stations to read. • To publish Web Logs (Blogs) via WAP or GPRS to the custommers of a telecommunication company • Publications are delivered to application by RSS and server creates pages that are suitable for mobile stations to read.

19. Company vision • capture a larger market share • become more attractive to customers by • developing better software quality products • managing better software development processes.

20. Project A Used techniques: • Feature Driven Development • Small team • Osmotic communication • Spiking, Walking skeleton • Information radiators • Simple design • Access to expert user • Estimation using comparison and analogy • Frequent delivery • Documentation is not so important • Kick-off meeting • Status reporting

21. Project A Home ground polar chart

22. Project A Framework for project assessment Included in the experience base

23. Project B Used techniques: • Feature Driven Development • Small team • Osmotic communication • Spiking, Walking skeleton • Information radiators (cont.)

24. Project B Used techniques: • Simple design • Acess to expert user • Estimation using comparison and analogy • Frequent delivery • Kick-off meeting • Status reporting • Code reviews • Measurements of quality characteristics • Burn charts

25. Project B Home ground polar chart

26. Project B Framework for project assessment:

27. Project A Quality Model Dark colors represent quality characteristics of the project mm n

28. Project B Quality Model Remark:We have two quality models,Project A and Project B. We just have to find a wayto measure it and prove that Project B is better.

29. Metrics and Characteristics • Project metrics: Personnel. • Product metrics: Lifetime, Downtime, Number of defects, Complexity. • Product quality metrics: Mean time to failure, Defect density, Customer satisfaction. • Other metrics: Lost amount of money to customer, Lost amount of money to developer. • Quantitative economic yields of improving software quality Remarks • Software quality metrics measured with Research Standard Metrics (RSM) tool • Quantitative economic yields of improving software quality are calculated (estimated) using a value-based framework.

30. RSM quality report

31. Project basic data (RSM Reports)

32. Software quality report (1):Total quality notices

33. Software quality report (2):Maintainability(Readability)

34. Software quality report (3):Quality notices

35. Software quality notices

36. Quantitative yields of improving software engineering methods Stoica, A.J. (2007). Exploring Quantitative Yields of Software Engineering Methods in the Software Life-Cycle. The first IEEE Computer Society EQUITY International Conference. 19-21 March 2007, Amsterdam, the Netherlands.

37. Motivation The problem of exploring quantitative yields of a new (improved) software engineering method in the software life-cycle has to be addressed in a systemic manner • relevant metrics and models have to be selected • their interrelashionships have to be expressed in a formal manner to detect possible inconsistencies and/or incompletenesses. The Model Systems (MS) concept is applied to • analyze multiple facets of software development • derive a value-based framework (VF) for exploring the quantitative yields of a new (improved) software engineering method in the software life-cycle.

38. Value-Based Framework (VF) Interrelated software system metrics and models that address: • Costs (CM) • Quality (QM) • Productivity (WM) • Benefits (BM) from applying better software project quality management • Resulting value estimation metrics (VM), evaluated over the software life-cycle.

39. Quantitative Models for Estimating Costs, Benefits, and Value Applicable for software engineering methods that: • Improve software quality via structured and facilitated group reviews • Certify companies to adhere to regulations for international trade • Evaluate, qualify, certify suppliers to government agencies • Perform individual and group-based project management via technical challenging approaches • Use sets of criteria designed for supplier selection and establishment of priorities for internal software process improvement.

40. VF Instantiation Example • Improving an agile (hybrid) software engineering method is achieved as a result of applying ISO/IEC software quality (SQuaRE) standards • Metrics and models are applied to: i) existing, and ii) new (improved) software development process in order to calculate all the possible quantitative yields • Improvements to an agile (hybrid) software engineering process are in similar to applying Software Inspection Method (SIM) • Guidance of practical use that ISO/IEC SQuaRE quality standard contains is also quite similar.

41. Input parameters (Project B)

42. General metrics

43. Cost metrics

44. Benefit metrics

45. Value metrics

46. Main results • The real return on investment is 4700%. • The real gross benefit is 49701,6 Eur. • Real net benefit when you apply Software inspection Method (SIM) 48666,15 Eur. • Real benefit to cost ratio i.e. the ratio between benefits and costs is 48. • Real breakeven point is 24288,90 Eur, or 202,12 hours.

47. Summary • Our research shows that there are no conflicts between applying ISO/IEC SQuaRE software quality standards and agile software development • Economic and quality benefits are demonstrated using a value framework as well as software metrics and measurements • Simple guide to SQuaRE practical use with real-life example is presented • Evaluation framework for software system projects to decide on the balance between agile and plan-driven approaches is outlined • Experience base for analysing software system projects is created • Our work is a step forward in reconciling agile approaches and formal software product quality evaluation that is discussed in ISO/IEC software quality standards.

48. Conclusions • We showed that creatively using software quality standards in agile software development is possible for • combining the best of agility and discipline • creating quality software • generating economic benefits. • During the last years, agile software development has become a work facet in small and medium software development organizations, but its lack of theoretical foundations require practical methods, models, metrics, and/or processes to relate it to software engineering research and application as well as to the associated engineering standards. • Our work supports this effort and offers theoretical and practical support for future advances in educational and software development organizations.