SWE 6763 – Project Presentation

Project Title: Software Product Quality metrics: A Software Quality Model and Metrics for assessing quality and identifying risks The effects of Development Team skill on Software Product Quality By: Radhika Godla Southern Polytechnic State University Spring 2011 Presented to: Dr. Frank Tsui School of Computing and Software Engineering SWE 6763 – Project Presentation

This paper will look at a two perspectives on software quality. Firstly, it presents the Software Assurance Technology Center's (SATC) model for software quality, developed at GSFC to meet the needs of NASA software project and explains a Software Quality Model and then uses it as a basis for discussions on quality attributes and risks. The quality attributes are used to derive a core set of metrics relating to the development process and the products, such as requirement and design documents, code and test plans. Secondly, this paper provides an analysis of the effect of the skill/experience of the software development team on the quality of the final software product. The second section of the paper provides an analysis of the effect of the skill/experience of the software development team on the quality of the final software product. A method for the assessment of software development team skill and experience is proposed, and derived from a workforce management tool currently in use by the National Aeronautics and Space Administration.

Software is composed of both high and low-level components. Through these components, and through the ways they are composed, software exhibits properties that characterize and distinguish it from other artifacts. Some of the properties of software are desirable. We call these desirable properties, qualities or quality attributes. A set of quality attributes is the vehicle through which the different interest groups express their needs of software. A goal-directed approach to building a quality model for software is effective for accommodating and balancing the needs of these different interest groups. The set of desirable properties or quality attributes of software provides an abstract or high-level specification for what we will call software product quality. Quality attributes we associate with software correspond either to behaviors or uses. A behavior is something that the software itself exhibits when it executes under the influence of a set of inputs (eg., reliability and efficiency are behaviors). From a functional perspective, a behavior of a software system can be characterized by the set of responses (including outputs and metrics) the system exhibits through the execution and interaction of sets of its functions in response to one or more sets of inputs. A use is something that different interest groups do with or to software (eg., portability and maintainability are uses). For example, A use is a user-performed function where the software is the input. Output <--- User-performed-function (software). A Software Characteristics is an abstract property (determinable) of software that classifies a set of tangible quality-carrying product properties. It is not a behavior or a use. Modularity is an example of a software or product characteristic. Software characteristics may correspond either to a set of functional entities or a set of non-functional tangible properties. Software characteristics help software to satisfy quality attributes. For example, tangible machine independent properties of software components contribute to a software system's portability. It follows that software characteristics may be used to support the definition of high-level quality attributes. Terminology and Framework:

SoftwareQuality models

As part of the mission to improve the quality of NASA software, the SATC is assisting software managers in establishing metrics programs that meet their needs with minimal costs, and interpreting the resulting metrics in the context of the supported projects. Their experience indicates that the pragmatic view of software quality cannot be evaluated using only software product goals and attributes; goals and attributes for the development process through the life cycle must also be evaluated. Following the structure of ISO 9126, the SATC model defines a set of goals that are important in the GSFC environment. The goals are then related to software product and process attributes that allow indications of the probability of success in meeting the goals. A set of metrics is chosen or developed that measure the selected attributes. From the pragmatic description of software quality, four goals are derived: Requirements Quality Product Quality Implementation Effectivity Testing Effectivity SATC Software Quality model

SATC Software Quality Model

The objectives for this first goal are complete, unambigious and understandable requirements document, the stabilization of requirements as quickly as possible, and the traceability of all requirements from their source to the software requirements document and then through design and implementation and test. The associated quality attributes are: Ambiguity – Requirements with potential multiple meanings Completeness – Items left to be specified. Understandability – The readability of the document. Requirement Volatility – The rate and time within the life cycle changes are made to the requirements Traceability – The traceability of the requirements upward to higher level documents and downward to code and tests. Requirements Quality Attributes and Metrics

An important objective of a software development project is to develop code and documentation that will meet the project's “utility” requirements. The specific attributes measured are as follows: Structure/Architecture: The evaluation of the constructs within a module to identify possible error-prone modules and to indicate potential problems in usability and maintainability. Reuse: The suitability of the software for reuse in a different context or application. Maintainability: The suitability of the software for ease of locating and fixing a fault in the program. Documentation: The adequacy of internal code documentation and external documentation. The attributes of Structure/Architecture, Reuse and Maintainability use the same metrics for evaluation, but with different emphasis. The three areas of metrics applied here are complexity, size, and the correlation of module complexity with size. Product Quality attribute and metrics

Complexity Measurements: It is generally accepted that modules with higher complexities are more difficult to understand and have a higher probability of defects than modules with smaller values. Thus complexity has a direct impact on overall quality and specifically on maintainability and re-useability. Projects developing code designed for reuse should be even more concerned with complexity since it may later be found to be more expedient to rewrite highly complex modules than to reuse them. There are many different types of complexity measurements, for example: Logical (Cyclomatic) Complexity: Number of linearly independent test paths Data Complexity: Data types and parameter passing Calling Complexity: Calls to and from modules GOTO Usage: Count of number of GOTO statements Nesting levels: Depth of nesting a conditional statement within a conditional statement. Logical (Cyclomatic) Complexity - The most common measure of the logic of a program is the Cyclomatic complexity (v(G)) that was proposed by McCabe. This metric measures the number of linearly independent paths through a program, which in turn relates to the testability and maintainability of the program. GOTO Usage: GOTO usage generally increases the complexity of a program because it disrupts the intended or normal structured flow of the program through the application of an unconditional branching instruction. The use of large number of GOTOs is a typical cause of producing unreadable programs with “spaghetti code”. A high count of GOTOs generally indicates lower quality code.

Size Measurements: One of the oldest and most common forms of software measurement is size. There are many possibilities for representing size, but I will be discussing only some common ways of counting lines. Types of line counts include: Lines of Code (LOC), Non-comment Non-blank (Source lines of code – SLOC), Executable statements. Size of modules is a quality indicator. General industry standards suggest that 50 to 100 lines is the maximum size that any module should attain; larger modules tend to be difficult to understand and thus lower in maintainability and reusability. Correlation of Complexity and Size: While both size and complexity are useful metrics, the correlation of the two produces even more information. The SATC has found that much of the code produced at GSFC has a linear relationship between complexity and size, such that the complexity of a “normal” module can be predicted by its size. Thus a reduction in size will reduce complexity. The use of this correlation is important in risk determination.

The second section of this paper provides an analysis of the effect of the skill/experience of the software development team on the quality of the final software product. A method for the assessment of software development team skill and experience is proposed, and derived from a workforce management tool currently in use by the National Aeronautics and Space Administration. Using data from 26 small-scale software development projects, the team skill measures are correlated to 5 software product quality metrics from the ISO/IEC 9126 Software Engineering Product Quality standard. An analysis of the degree of correlation between the development team skill measures and the software product quality measures is presented. II. The effects of development Team Skill on Software Product Quality

Measurement of Development Team Skill:The assessment of the capability of the software development team requires insight into the team's collective education and experience. The assumption is that a more educated and experienced team will produce higher quality in their software product and artifacts. The construction portion of the software life cycle is comprised of three phases, each which requires different types of skills to accomplish successfully. • The requirements phase is focused on organizational and interpersonal skills; interacting with customers, eliciting needs from those customers, and organizing the needs into a set of requirements. • The design phase requires the ability to create a software solution that meets the identified requirements, and skills in evaluating the set of approaches and tools to implement that solution. • Implementation demands a working knowledge and skill in the specific development tools, including languages, operating systems, libraries and development environments. While integration and test are a vital part of the development life cycle, the focus of this study is on skill and experience in the “building” phases of development.

Phase Skill Level Measure Description Requirements Level 1 Proportion of the development team involved in requirements development that was assessed at Skill Level 1. Level 2 Proportion of the development team involved in requirements development that was assessed at Skill Level 2. Level 3 Proportion of the development team involved in requirements development that was assessed at Skill Level 3. Level 4 Proportion of the development team involved in requirements development that was assessed at Skill Level 4. Design Level 1 Proportion of the development team involved in software design that was assessed at Skill Level 1. Level 2 Proportion of the development team involved in software design that was assessed at Skill Level 2. Level 3 Proportion of the development team involved in software design that was assessed at Skill Level 3. Level 4 Proportion of the development team involved in software design that was assessed at Skill Level 4. Implementation Level 1 Proportion of the development team involved in software construction that was assessed at Skill Level 1. Level 2 Proportion of the development team involved in software construction that was assessed at Skill Level 2. Level 3 Proportion of the development team involved in software construction that was assessed at Skill Level 3. Level 4 Proportion of the development team involved in software construction that was assessed at Skill Level 4.

Measurement of Software Quality: Software product quality is captured using the ISO/IEC 9126 as a guide for the expectations of quality within the delivered software product. The ISO/IEC 9126 is an international standard for software product quality that represents the quality of a delivered software product in terms of six major characteristics: Functionality, Efficiency, Reliability, Usability, Maintainability and Portability. Table 4: Software Product Quality Metrics

Experimental Approach:The approach to establishing the relationship between development team skill and measures of software quality was to record skill and quality measures from actual software development projects and analyze the results. A correlation coefficient was calculated to quantify the relationship between each of the skill variables, and each of the software product quality metrics. Equation 1: Correlation Coefficient Calculation

Using the measurement frameworks outlined above, it is appropriate to hypothesize the results of the analysis in terms of correlating development team skill to software product quality. Table 5 captures the expected relationship between the Development Team skill levels and the software product quality metrics. The expectation of a direct relationship between the increasing skill levels and the increasing correlation is annotated in Table 5 as “Incr.”. For example, as the design skill increases, it would be expected that the correlation between that skill value and the Design Adequacy metric would increase. This indicates a positive relationship and the influence of the higher level of skill and experience on the design of the system. The “Decr.” annotation captures the expectation that an inverse relationship, or decrease in correlation, exists for the increasing skill levels. Requirements Volatility provides a good example of this case. As requirements skill increases, it is intuitively expected that the volatility of the requirements will decrease. Table 5: Expectations for correlating skill to Software Quality

Effects of skill on adequacy :As expected, the increased presence of more highly skilled software developers on the development team was positively correlated with the quality measured in the final software product. This was most evident in the correlation results of various skill levels to the measures of Design and Implementation Adequacy. There is a sharp contrast between the correlation of the highly skilled (Levels 3 and 4) developers to adequacy, and the correlation of the marginally skilled (Levels 1 and 2) developers to Adequacy. The increasing presence of the less skilled in each phase of the development life cycle has a significant negative correlation to the adequacy of the design and implementation. The practical inference made from these relationships is that increasing numbers of less skilled developers decrease the quality of the software product in terms of its adequacy. Conversely, the increasing presence of more skilled developers has a significant positive correlation to the adequacy of the software product, implying that increasing numbers improve the quality of the final product. Effects of skill on specification Volatility: The correlation coefficients associating development team skill to Functional Specification Volatility were as expected in terms of their inverse relationship. That is, skill levels increased, there was a general decrease in the volatility of the functional specification. Despite the expected behavior, the magnitude of the correlation coefficients is surprisingly low. Intuitively, one would expect volatility to decrease dramatically as the collective skill of the requirements team increased. The low correlation coefficients indicate that requirements skill has very little impact on the volatility of the software specification. Adverse Impact of skill Level 2: The most prominent unexpected result is the divergence of skill Level 2 from the expected increasing and decreasing correlation patterns. For each quality attribute, and particularly with respect to design and implementation skill, the increased presence of personnel with Skill level 2 has an adverse impact on that quality attribute. The unexpected adverse impact of skill Level 2 may be the result of a flaw in the assignment of skill levels to personnel. Another explanation is that the deviation of Skill level 2 is simply an indication that the presence of less skilled developers has a negative influence on a development effort.

Effects of Requirements Skill: The effect of requirements skill in software product quality was most pronounced in its effect on Design and Implementation Adequacy. The presence of higher requirement skill levels had a strong correlation to the adequacy of the design and implementation, while the presence of lower requirement skill levels had an opposite and negative effect on adequacy. Increasing numbers of engineers that are unskilled in requirements development appear to impair a project's ability to provide adequate design modules and source code units. This results supports the notion that the quality of the requirements has a dramatic effect on its downstream products in the design and implementation phases of development. In analyzing the effect of requirements skill on the Functional Implementation Coverage and Completeness, it is interesting to note the lack of significance of skill levels except for Skill Level 4. It appears that only the involvement of the most highly skilled requirements engineers has a significant effect on the correctness and completeness of the final software product. This may be an indication of leadership in developing requirements development team. Effects of Design Skill: The design skill measures correlated most significantly with the adequacy of the design and implementation. This aligns with the intuitive assumption that those with a higher level of design skill and experience will produce a more adequate design, and therefore a more adequate implementation and vice-versa. Effects of Implementation Skill: The measures of Implementation skill correlated most significantly to implementation adequacy. As with the design and requirements skill, increasing numbers of less skilled developers were negatively correlated to Design and Implementation Adequacy, and increasing numbers of more skilled developers were positively correlated to those quality attributes. Implementation Skill Level 4 had a moderate correlation to the functional implementation completeness and coverage metrics.

Conclusion:The first section of this paper discusses about the SATC's model for software quality, developed at GSFC to meet NASA's mission to improve the quality of NASA software. The Software Assurance Technology Center (SATC) has applied some of the concepts from the theoretical models of software quality to develop a unique model that fits the needs of project managers in the NASA and GSFC environment. The model is comprehensive, starting with specifying goals through presenting the results. It is also applicable across the development life cycle adapting the goals to the life cycle phase based on the project's specific objectives. Currently, software managers and quality assurance engineers are starting to budget for data collection and software metric analysis. Significant effort is being spent on methods to use the metrics to forecast the values of the selected goals and attributes forward to project milestones such as delivery of the software. Metrics introduced by the SATC, especially those relating to requirements quality need to be validated, and a set og higher level design quality metrics is needed to supplement those done at the code level. The company's long term objective is to be able to establish a numerical metric scale for assessment of all of the metrics that affect software quality. The second section of this paper provided empirical evidence that development team skill has a significant effect on the quality of a software product. A measurement framework for assessing software development skill and experience was used to quantify the capabilities of the software development team. These measures were correlated to measures of software product quality taken from the ISO/IEC 9126 standard. The proposed framework for measuring development team skill and experience provided an adequate representation of the distribution of individual skills on a given development team. It is recommended that software project managers assess their software developers using the proposed framework in order to better understand the strengths and weaknesses of their staff. Such an understanding could prevent the assignment of tasks that are not well suited to less skilled and experiences engineers, and thus prevent the negative impact on product quality.

References: • http://satc.gsfc.nasa.gov/support/STC_APR96/qualtiy/stc_qual.html • http://www.sei.cmu.edu/reports/88cm012.pdf • The Effects of Development Team Skill on Software Product Quality: By Justin M. Beaver and Guy A. Schiavone – ACM Software Engineering Notes Volume 31 • http://www.sqi.gu.edu.au/docs/sqi/misc/SPQ-Theory.pdf

SWE 6763 – Project Presentation