Chapter 3 - A goal-based framework for software measurement

1. Chapter 3 - A goal-based framework for software measurement

2. Objectives • To understand the fundamental classes of software measures • To understand the difference between internal and external attributes • To understand the Goal-Question-Metric (GQM) approach

3. Classifying software measures • In software engineering, there are three fundamental classes of entities to be measured: • Processes – collections of software-related activities. • Products – any artifacts, deliverables or documents that result from a process activity. • Resources – entities required by a process activity. • The activities in the process are ordered or related in some way that depends on time, so that one activity must be completed before another can begin. • The timing can: • Explicit, as when design must be complete by October 31 • Implicit, as when a flow diagram shows that design must be completed before coding can begin.

4. Classifying software measures (cont’d) • Resources and products are associated with the process. • Each process activity has resources and products that it uses, as well as products that are produced. • The product of one activity may feed another activity. • For example, a design document can be the product of the design activity which is then used as an input to the coding activity.

5. Classifying software measures (cont’d) • Within each class of entity we have internal and externalattributes. • InternalAttributesof a product, process or resource are those that can be measured purely in terms of the product, process or resource itself (separate from its behavior). • ExternalAttributesof a product, process or resource are those that can be measured only with respect to how the product, process or resource relates to its environment (the behavior is important rather than the entity itself).

6. Classifying software measures (cont’d) • For a software module, some internal attributes can be: • Size (lines of code) • Cyclomatic complexity. • Dependencies among modules (coupling). • We call these internal attributes because we can measure them without executing the code. • On the other hand there are some external attributes that can’t be measured without executing the code. For instance: • Number of failures experienced by the user. • Difficulty faced by the user in operating the program. • Length of time it takes to search the database.

7. Classifying software measures (cont’d) • Managers often want to be able to measure and predict external attributes, for example: • Cost-effectiveness of an activity • Productivity of the staff • Users are also interested in external attributes, for example: • Reliability • Usability • Portability • However, external attributes are usually more difficult to measure than internal ones, and they are measured quite late in the development process. • For example, reliability can be measured only after development is complete and the system is ready for use.

8. Goals of software-measurement research • Identify relationships among internal and external attributes • Find new and useful methods for measuring directly the attributes of interest

9. Process measurement • Measurement can help us answering questions about our process: • How long it takes for a process to complete? • How much it will cost? • Is it effective or efficient? • How it compares with other processes that we could have chosen? • Only a limited number of process attributes can be measured directly. These measures include • The duration of the process • The effort associated with the process • The number of incidents of a specified type arising during the process • The external process attributes are usually measured using some directly measurable process attributes

10. Example: inspection effectiveness • In AT&T, managers needed to evaluate the cost of the inspections against the benefits received (cost-benefit analysis) • The average amount of effort expended per KLOC reviewed • Inspection effectiveness • Average faults detected per KLOC

11. Example: testing process • The testing process may be composed of unit testing, integration testing, system testing, and acceptance testing • Each component process can be measured to determine how effectively it contributes to overall testing • We can track the number of errors identified in each sub-process, along with the duration and cost of identifying each error, to see if each sub-process is cost-effective

12. Sample Process Measures

13. Product measurement • What is a product? • Products are not restricted to the items that management is committed to deliver to the customer. • Any artifact or document produced during the software life cycle can be measured and assessed.

14. External product attributes • Since an external product attribute depends on both product behavior and environment, each attribute measure should take these characteristics into account. • For example, if we are interested in measuring the reliability of code, we must consider the machine on which the program is running as well as the mode of operational usage. • Usability, integrity, efficiency, testability, reusability, portability and interoperability are other external attributes that we can measure.

15. Internal product attributes • Internal product attributes are sometime easy to measure. • For example, we can determine the size of a product by measuring the number of pages it fills or the number of words it contains. • Other internal product attributes are more difficult to measure, because opinions differ as to what they mean and how to measure them. For example, there are many aspects of code complexity.

16. The importance of internal attributes • Users sometimes dismiss many of these internal attributes as unimportant, since a user is interested primarily in the ultimate functionality, quality and utility of the software. • However, the internal attributes can be very helpful in suggesting what we are likely to find as the external attributes. • A major reason that developers want to use internal attributes to predict external ones is the need to monitor and control the products during development. • For example, knowing the relationship between the internal design attributes and failures, we want to be able to identify modules at the design stage whose profile, in terms of measures of internal attributes, show that they are likely to be error-prone or difficult to maintain or test later on.

17. Sample Product Measures

18. Resource measurement • Resources to be measured include any input for software production. • Personnel – individual or teams. • Productivity of personnel. • Material– software and hardware tools. • Effectiveness, user-friendliness, understandability, etc., of the tools. • Methods – languages, design styles, coding styles, etc. • Effectiveness and suitability of methods. • We measure resources to determine their: • Magnitude (how many staff are working on this project?) • Cost (how much are we paying for testing tools?) • Quality (how experienced are our designers?)

19. Sample Resource Measures

20. Goal-Question-Metric (GQM) • To use Goal-Question-Metric: • Identify the overall goals of your organization. • Generate questions whose answers you must know in order to determine if your goals are being met. • Analyze each question to identify measurements you need in order to answer each question.

21. Goal-Question-Metric (GQM) (cont’d) • Goal-Question-Metric paradigm has been proven to be effective in selecting and implementing metrics • The GQMapproach provides a framework involving three steps: • List the major goals of the development or maintenance project. • Derive from each goal the questions that must be answered to determine if the goal is being met. • Decide what must be measured in order to be able to answer each question.

22. Goal evaluate effectiveness of coding standard Questions Who is using standard? What is coder productivity? What is code quality? • Proportion of coders • Using standard • Using language • Experience of coders • With standard • With language • With environment, etc. • Code size • Lines of code. • Function points, etc. Effort Errors Metrics Example 1: GQM tree on evaluating effectiveness of coding standard.

23. Example 2: GQM tree on assessing inspection process

24. Example 3: GQM tree on software reliability

25. Example 4: GQM tree on improving software maintainability

26. Templates for goal definition • Purpose: To (characterize, evaluate, predict, motivate, etc.) the (process, product, model, metric, etc.) in order to (understand, assess, manage, engineer, learn, improve, etc.) it. • Example: To evaluate the maintenanceprocess in order to improve it. • Perspective: Examine the (cost, effectiveness, correctness, defects, changes, product measures, etc.) from the viewpoint of (developer, manager, customer, etc.) • Example: Examine the cost from the view point of the manager. • Environment: The environment consists of the following: process factors, people factors, problem factors, methods, tools, constraints, etc. • Example: The maintenance staff are poorly motivated programmers who have limited access to tools.

27. Summary • All entities of interest in software can be classified as either processes, products or resources. • Attributes are either internal or external. • The Goal-Question-Metric paradigm is a useful approach for deciding what to measure. • The GQM approach creates a hierarchy of goals to be addressed, questions that should be answered in order to know if the goals have been met, and measurements that must be made in order to answer the questions.