Error reports as a source for SPI

Error reports as a source for SPI Tor Stålhane Jingyue Li, Jan M.N. Kristiansen IDI / NTNU

Goals Business: How can we reduce the cost of corrective maintenance? Research: What are the main cost drivers for corrective maintenance

Company A – 1 Company A is a software line company with only one product. The product is deployed on more than 50 operating systems and hardware platforms. The company has 700 employees and of these 400 are developers and testers.

Company A – 2 Via the company’s DTS – Defect Tracking System – the company collected: • Defect ID, priority, severity and report creation date • Defect summary and detailed description • Who found the defect and when • Estimated correction time • When was the defect corrected • Detailed comments and work log of the person who corrected the defect

What we did in company A We improved the company’s DTS based on the IBM concept of orthogonal defect classification – ODC. Based on a study of earlier reported defects, the classification scheme was modified to fit the company’s needs.

DTS system enhancements at company A

Company B – 1 Company B is a software house that develops business critical systems, primarily for the bank and finance sector. Most of their projects have a fixed price and a fixed delivery date. The company has 800 developers and testers.

Company B – 2 Via the company’s DTS the company collected among other things: • Defect ID, priority, severity and report creation date • Defect summary and detailed description • Who found the defect and when • Who tested the correction and how

What we did in company B Just as for company A, we improved the company’s DTS based on a study of earlier reported defects. In addition, the changes also enabled us to collect data that should later be used for software process improvement.

DTS system enhancements at company ¨B

Data collection In both companies, we collected defect data when the companies had used the new DTS for six months. Only defect reports that had all their fields filled in were used for later analysis. This gave us: • Company A – 810 defects • Company B – 688 defects

Data analysis – 1 Our goal was to identify cost drivers for the most expensive defects. Thus, we split the defects into categories depending on reported “time to fix”. • Company A – two groups: “easy to fix” and “time consuming” • Company B – three groups: “simple”, “medium” and “extensive”. “Simple” and “medium” defects was combined into one group – “simple” to be compatible with company A

Data analysis – 2 We identified the most important root causes for the costly fixes through qualitative analysis. • For both companies we had “correction type” and “cause”. • We also found important information in the • developer discussions (company A) • test descriptions (company B)

Root causes for high effort fixes - A

Root causes for company A Based on the collected data, we identified the most important root causes for costly defects: • It was hard to determine the defect’s location • Implemented functionality was new or needed a heavy rewrite • Long, costly discussion on whether the defect report really was a defect or just misuse of the system.

Root causes for high effort fixes – B

Root causes for company B Bad specifications and development problems account for 91% of the high effort defects. If we consider the sub-categories defined for these two root causes, we find that 70% of all correction costs are due to: • Errors in business logic • Unclear requirements • Problems in graphical interface

Important maintenance factors – 1 Several published studies have identified the following important maintenance cost factors: • Maintainers’ experience with system and application domain • System size and complexity • The development activity where the defect is discovered • Tool and process support

Important maintenance factors – 2 • System size and complexity – large complex systems makes it difficult to • Analyze the system to decide where the defect stems from - A • Deiced how to fix the defect - A • Find the right solution to implement – A, B

Important maintenance factors – 3 • Low maintainers’ experience with system and application domain cause • Need for heavy rewrite of functionality - A • Development problems, e.g. with business logic and user interface – B

Important maintenance factors – 3 ISO 9126 defines maintainability as: • Analyzability • Changeability • Stability • Testability The high maintenance cost factors size and complexity fit well into this model. However, the model focuses on software characteristics and ignore the influence of the developers’ knowledge and experience

Conclusions - 1 Important data sources when analyzing maintainability are: • Resources needed for fixing • Defect typology, e.g. ODC • Qualitative data such as test description and developer discussions The effort model should be updated regularly, based on the defect profile and project environment.

Conclusions – 2 There is no “best estimator” for corrective maintenance. Important factors are • Software characteristics – e.g. as defined in ISO 9126 • Staff characteristics – e.g. knowledge and experience

Error reports as a source for SPI