Objectives

1. Objectives • The aim of this topic is to introduce the concepts of measurement and metrics as well as the practical skills necessary to define and count LOC and FP, the basis for many software metrics. • You will: • Develop an understanding of why is it important to measure the process of software engineering and the products it produces • Develop practical skills in size measurement and basic function point analysis.

2. Introduction • Anything that you need to quantify can be measured in some way that is superior to not measuring it at all. Tom Gilb • When you can measure what you are speaking about and can express it in numbers, you know something about it. But when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind. Lord Kelvin • If software development is to be viewed as an engineering discipline, it requires a measurement component that allows us to better understand, evaluate, predict and control the software process and product. Victor Basili, University of Maryland

3. Why Measure? • We measure for a number of reasons. These include: • To characterize - to gain an understanding of products, processes,... • To evaluate - to determine status with respect to plans. • To predict - so that we may plan • To Improve - rational use of quantitative information to identify problems and strategies to remove them

4. What to Measure • Process • Measure the efficacy of processes. What works, what doesn't. • Project • Assess the status of projects. Track risk. Identify problem areas. Adjust work flow. • Product • Measure predefined product attributes (generally related to ISO9126 Software Characteristics)

5. The Goal Question Metrics (GQM) Approach • The GQM approach is based on the idea that in order to measure in a meaningful way we must measure that which will help us to assess and meet our organisational goals. • A Goalis defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view, relative to a particular environment. The objects of measurement are products, processes and resources. • Questions are used to characterize the way the assessment/achievement of a specific goal is going to be performed based on some characterizing model. Questions try to characterize the object of measurement with respect to a selected quality issue and to determine its quality from the selected viewpoint. Questions must be answerable in a quantitative manner. • Metrics are associated with every question in order to answer it in a quantitative way.

6. The Goal Question Metrics (GQM) Approach • Example: • Goal: To improve the outcome of design inspections from the quality managers point of view. • Question: What is the current yield of design inspections? • Metric: inspection yield = nr defects found / est. total defects • Question: What is the current inspection rate? • Metric: individual inspection rate, inspection period, overall inspection rate • Question: Is design inspection yield improving? • Metric: current design inspection yield / baseline design inspection yield * 100 • ...

7. Six Sigma Approach • What Is Six Sigma? • Six Sigma is a rigorous and disciplined methodology that uses data and statistical analysis to measure and improve a company's operational performance by identifying and eliminating "defects" in manufacturing and service-related processes. • Commonly defined as 3.4 defects per million opportunities, Six Sigma can be defined and understood at three distinct levels: • metric, methodology and philosophy...

8. Six Sigma Approach • In Six Sigmameasurement programs focus on collecting data that will help you to improve your processes in order to better satisfy your customers. • It is a common sense notion that something you do in working on a product (some part of the process) will have an effect in terms of the customer's perception of the quality of that product, and that by identifying problems with the process and making subsequent process improvements will improve the customer's perception of the product. • The measurments that you should be making are, therefore, the ones that will help you to assess and improve processes with the goal of satisfying the needs of the custmer(s).

9. Six Sigma Approach • This approach bases itself on the operational statement: you must state concisely what it is that you are going to improve (or achieve): "what's wrong with what". • It must be written in simple language using terminology that has the same meaning to everyone who is going to read it. • An operational statement will be stated in terms of the effect of what it is that you are improving. • It should aim to identify the where, when, what and how big of the problem.

10. Six Sigma Approach • For example: • the statement "the customers aren't happy" is of very little use in making them happy. • However, by repeatedly asking why and by analyzing any available data we might arrive at a concise statement such as "The XYZ module is unavailable 10% of the time". • This is a much more useful metric against which we could base the assessment of process changes. • To get to a statement like this requires measurementof availability.

11. Six Sigma Approach • In order to assess process improvement we must collect data over time. • This should begin with establishing baseline performance. A baseline is the average of historical data over a specified period of time. • In the example above, Module XYZ is unavailable 10% of the time is baseline a statement of the baseline performance. • Then we must then establish a goal. • This goal is stated as a measure of improvement over a period of time in terms of the baseline data. • The Six Sigma rate of improvement goal is often quoted as being a 10X improvement over a period of 2 years. If we take this improvement goal and apply it to the example above we would establish the 10X/2years improvement goal as Module XYZ is unavailable 1% of the time. Or stated more usefully, "Module XYZ is available 99% of the time". • So summarizing the example, we may have the following: • Operational Statement: "Module XYZ is periodically unavailable" • Historical Data: Availability metrics calculated for module XYZ over the past 6 months indicate that it is available 90% of the time. • Baseline performance: Module XYZ availability = 90% • 10x/2years improvement goal: Module XYZ availability = 99% • In order to verify our progress toward the goal we must track actual progress planning and performing periodical reviews of this performance

12. Process Metrics • Process metrics primarily focus on organizational performance and quality achieved as a consequence of a repeatable or managed process. This characterization and evaluation of a process for achieving performance and quality outcomes is known as Quality Assurance. • Process metrics include metrics relating to: • statistical SQA data • defect categorization & analysis • defect removal efficiency (DRE) DRE = E/(E+D) E number of errors before delivery (or before a particular phase) D number of defects after delivery • defect propagation from phase to phase • reuse data

13. Project Metrics • Software project metrics are used to adapt workflow and technical activities. • They are primarily used in estimation, monitoring and control of workflow within a project. • Effort/time per SE task • Defects detected per review hour • Scheduled vs. actual milestone dates • Changes (number) and their characteristics • Distribution of effort on SE tasks

14. Product Metrics • product metrics generally focus on the quality of deliverables. They include: • measures of analysis model • complexity of the design • internal algorithmic complexity • architectural complexity • data flow complexity • code measures (e.g., Halstead) • Measures of maintainability • defect metrics

15. Measurement Program • A measurement program is an effective means for controlling the software process performance (i.e., the actual results a project achieves by using its defined process) and guiding improvements in the software engineering processes. • It has to based on a model in which a comprehensive measurement framework may be constructed, analyzed, and modified as the organization’s goal change. • In order to define such a program, it is presented the underlying life cycle that • applies to the use of software metrics.

16. Metrics Life Cycle • Planning: to identify the scope of the measurement program through • Metric selection • Identify the measures to support organization objectives. The use of Goal/Question/Metric paradigm ensures that all the strategic goals of the company are measured. • Metric specification • Define the operational measurements procedures: selected metrics have to be provided with specific meanings so that these can be uniformly collected and interpreted

17. Metrics Life Cycle • Implementing: data are collected in accordance with the operational measurement procedures, validated, and analyzed. • Data collection • apparently the most easy, but also the most important. • To be effective it must be automated. • Data validation • one of the easiest and most cost effective way to validate data is simply to observe the data collection process and make sure that data are collected in accordance with the defined procedure (to ensure at least standardization)

18. Metrics Life Cycle • Data analysis • to evaluate the status of the project with respect to plans • to improve the software process • In processanalysis allows to evaluate whether the project is drifting off track, so that they can be brought back under control by suitable actions. • A posteriorianalysis permits to controls the trend of the key measures of the whole company, and to refine the baselines to compare against, so that it can be judged whether or not the improvement actions are working as intended and what the side effects may be.

19. Metrics Life Cycle

20. Metrics Life Cycle • Improving: periodically, the entire measurement program is to be reviewed to ensure that it helps the software projects to control their processes, and the whole organization to improve. Metrics can be revised, redefined following again the metric life-cycle. • In addition, in case there exist metrics with dubious values or not used, they can even be retired.

21. Metric Objective • The main purposes of measuring are: • • measuring the Productivity of a project • • measuring and evaluating the Quality of the software product • • controlling a project. • in the SEI CMM Level 3 the management and control of size/complexity, reusable software components, effort and costs is required. • These are also the goals to be used in the GQM approach.

22. Productivity • Establish the relationship between the output and the input of a project • Problems: • • establishing a measure unit for the output • • recognizing all the project/product/management characteristics affecting the expended effort • • measuring the other factors affecting the effort, as rework due to change and reuse

23. Quality • In order to measure the quality of a software product aspects such as : • The defects density and rate • The removal efficiency • The number of bad fixes • have to be considered.

24. Controlling the project • As a support for the Project manager to provide information for performing as much precise estimate as possible and for supporting technical and managerial choices. • problems to be resolved controlling a project are related both to prediction and evaluation of • effort • scheduling • stability • defect

25. Primary Data • The Primary Data are the lower level of the information to be collected and documented when prescribed, in order to allow the calculation analysis helpful to monitor the process and evaluate the products. • These are: • Size • Effort • Defects • Duration (estimated and actual) • Stability (Requirements and Staff Changes)

26. Size • Furthermore the size represents the starting point for the whole metrics system. • It is involved in both assessment and predictive metrics; • for assessment, it is used to measure an artifact or a system, or also to normalize other metrics • for prediction it serves to give a concrete mean to express the provisions: the strong connection between Product size and the effort needed to develop the Software Product, the size allows to derive accurate estimates of the effort, cost, and schedule of the project

27. Size • The product size can be expressed in different unit of measurement, according to the adopted estimating/ measuring technique. • As a Backfiring method exists for determining the correspondence between number of Function Points and number of LOC so it is • possible to establish a correspondence factor between Object-Oriented Function Points and Function Points

28. Size – Arpad method • Measures performed at the beginning of the Concept Exploration can only provide indicative information (low confidence level) on the target product size and can be gathered only with indirect measures; the calculation is based on the estimate of the effort; the obtained measure can have a low confidence level, • This method is based on the computation of a quantity corresponding to the effort needed to the specification of the system. • This calculation considers factors as • number of people to be interviewed, • number of days required to the interviews, • number input and output screens, reports, menus and so on. • With a very simple step, it is possible to obtain a first, indicative estimate of the global effort needed for the complete development of the Software Product (under certain hypothesis.) • Considering a reference Productivity Parameter the estimated global effort allows to derive a first estimate of the size.

29. Functionality Oriented Measures (Function Points) • A direct estimate of the size is possible at the end of the Concept Exploration, using the Function Point Analysis method based upon the artifacts produced, i.e. System Requirements and System Architecture. • The Function Point Analysis method gives a way of sizing software through the analysis of the implemented functionality of a system from the user‘s point of view, considering the application as a black box.

30. Functionality Oriented Measures (Function Points) • It consists, simplifying, of two main steps: • to evaluate the unadjusted Function Points (UFP) count, basing on the number of user functions and on its complexity classification • to evaluate the adjusted Function Points (FP) count, by scoring fourteen general system characteristics. • These steps allow the calculation of the product size keeping into account also its complexity, i.e. the degree of complicatedness of the product owing to the structure of the aggregated components and related relationships.

31. Function Points comp. • STEP1 • Count the number of external inputs, external outputs, external inquiries, internal logic files, and external interface files required. The IFPUG provides the following definitions for these: • External Input (EI): An EI is an elementary process in which data crosses the boundary from outside to inside.  This data may come from a data input screen or another application. The data may be used to maintain one or more internal logical files.  The data can be either control information or business information.  If the data is control information it does not have to update an internal logical file. • External Output (EO): An EO is an elementary process in which derived data passes across the boundary from inside to outside.   Additionally, an EO may update an ILF.  The data creates reports or output files sent to other applications.  These reports and files are created from one or more internal logical files and external interface file. • External Inquiry (EQ): An EQ is an elementary process with both input and output components that result in data retrieval from one or more internal logical files and external interface files.  The input process does not update any Internal Logical Files, and the output side does not contain derived data. • Internal Logic File (ILF): AN ILF is a user identifiable group of logically related data that resides entirely within the applications boundary and is maintained through external inputs. • External Interface File (EIF): An EIF is a user identifiable group of logically related data that is used for reference purposes only. The data resides entirely outside the application and is maintained by another application. The external interface file is an internal logical file for another application.

32. Function Points comp. • STEP2 • Rate each component (EI,EO,EQ,ILF,EIF) as low, average, or high. • For transactions (EI’s, EO’s, EQ’s) the ranking is based upon the number of files updated or referenced (FTR’s) and the number of data element types (DET’s). • For both ILF’s and EIF’s files the ranking is based upon Record Element Types (RET’s) and Data Element Types (DET’s). • A record element type (RET) is a user recognizable subgroup of data elements within an ILF or EIF. • A data element type (DAT) is a unique user recognizable, non-recursive, field. • File Type Referenced (FTR): A FTR is a file type referenced by a transaction. An FTR must also be an internal logical file or external interface file.

33. Function Points comp. • STEP3 • Multiply each count by the numerical rating shown for (low,average,high) to determine the rated value. • Sum the rated values in each row (EI,EO,EQ,ILF,EIF) giving a total value for each type of component . • Sum the totals in the rightmost column for each component to give Total Number of Unadjusted Function Points (UAF).

34. Function Points comp. • STEP4 • Next we calculate a value adjustment factor (VAF) based on 14 general system characteristics (GSC's) that rate the general functionality of the application being counted. • Each characteristic has associated descriptions that help determine the degrees of influence of the characteristics. The degrees of influence range on a scale of zero to five, from no influence to strong influence.

35. Function Points comp. • STEP5 • Calculate the Value Adjusted Function point (VAF) using the IFPUG value adjustment equation with the 14 weightings allocated to each General System Characteristic (GSC): • VAF = 0.65 + 0.01*Sum(GSCi) • for all i in [1..14], where 0.65 and 0.01 are empirically derived constants. • That is, VAF is calculated by summing all of the answers to the 14 questions (weighted 0-5), multiplying this sum by the factor 0.01 and then adding 0.65. • STEP 6 • The final Function Point Count is obtained by multiplying the VAF times the Unadjusted Function Point (UAF). • FP = UAF * VAF (function points)

36. Product size measures at the Design phase • As the design describes directly the selected solution of the problem, it contains precise information and Function Points method can be usefully applied at this level. • At this level, where the use of reuse and COTS are well defined and delimited, detailed estimates are possible as only those classes to be developed are included in the count. • So the used method could be: • Determination of LOC from class/method analysis, backfired to FP • OOFP calculation converted to FP

37. OOFP • The Object-Oriented Function Points method uses an object-oriented specification, focusing on object, attributes, and operations. Applying this method, the boundary can be moved to surround individual classes; in this way not only delivered functionality are measured, but also the size and the complexity of the application. • When dealing with OO Programming, these steps have to be ensued • Look at each cluster of object classes as a system and count Function Points • Analyze the Object Diagrams and count: • Data Items (Internal Data Items, External Data Items) • Service Requests (Incoming, Outgoing, Status Inquires) • Forget value adjustment factors • Report unadjusted Function Points

38. Product oriented measures (LOC) • At the beginning of the construction/module test phase, an estimation of the product size, in terms of Lines of Code, has to be performed. • by means of the backfire method, which derives the LOC from the counted FP • or using corporate data

39. What’s a LOC • The most general approach to size measurement is to count the number of text lines in a source program. In doing this, we typically ignore blank lines and lines with only comments. All other text lines are counted. This approach has the advantage of being simple and easy to automate. • This LOC counting approach has the disadvantage of being sensitive to formatting. Those programmers who write very open code will get more LOC for the same program than would their peers who used more condensed formats. • Even comments influence this counting, you can strip comments, but commented code is much more maintainable than uncommented one. • In doing this, you should also establish the practice of putting a logical LOC on each physical line of the source program.  

40. Statements Count (logical LOC) • Counts of logical statements attempt to characterize size in terms of the number of software instructions, irrespective of their relationship to the physical formats in which they appear • Either the operational definition of LOC and statement is to be provided through tables that explicitly identifies the values for each attribute that is to be included in or excluded from our statement counts. • Examples of such table are from • Robert E. Park “Software Size Measurement: A Framework for Counting Source Statements” CMU/SEI-92-TR-020

41. Size versus development time • LOC are precise, machine countable, environment specific, and they can reflect the mix of reused, modified, and new code. • The principal disadvantage is that LOC are hard to visualize early in a project. • the size and development time for the C++ programs I have developed are highly correlated. • This means that once I have estimated the size of a new program, I can readily calculate the time it will take me to develop it.

42. Size versus development time • But .. • What about the use of other languages (other than C++)? • What about other development groups (the experience of developers is important) ? • What about maintenance (there is not just development)? • What about the quality of requirements, architecture, design, ….. ? • Then ….

43. Size versus development time • This can be true for just a company, • steady (without great turn-over) • with only a particular kind of software developed (unless it develops metrics for every kind of sw) • just in one language (unless having metrics for all languages) • Which collects historical data to track its productivity that will be used for forecasting new products costs

44. Backfiring • A correspondence between the number of LOC and Function Points was researched by Capers Jones, resulting in a technique called Backfiring • This technique makes available a table that, for each different language, provides the number of LOCs corresponding to a Function Points. • But • the margin of error in converting LOC data into Function Points or back is high • the Backfiring conversion factors have to be derived by studying a large number of products, belonging to different domains

45. Productivity • The productivity is generally measured as the number of working hours needed for the production of a single unit. • Simple? Yes, but …. • Many productivity indicators exist in literature; these are generally obtained as mean values of a set of different projects, which were developed by people with different skills, working in different development environments. • Many factors which influence these indicators exist. They are not present in every organization at the same time, neither standardizable. • Each organization must collect its own data and use them for the productivity evaluation, concentrating only on factors allowed by the available data. • The level to which these factors are present differ from one organization to another, therefore it is important to collect and adopt a company’s own data in different periods. The best way to handle the productivity function is to base it on a family of factors.

46. Productivity • Productivity is affected by: • Project characteristic: describing people and processes involved in the development effort. • Management characteristic: describing the way how a project is managed. • Product characteristic: reflecting the nature of the product itself.

47. Productivity • Project characteristic • Personnel: the topic of personnel includes most of the “human elements” in a project. • The personnel characteristic are listed below: • Education: the level of degree • Experience: the numbers of years of relevant experience in software engineering, or with • specific software application and technologies or in the organization. • Expert Assistance: it refers to a person (or group of person) external to a project team who • has knowledge and experience in the application being developed by the project team. • Training: external or internal. • Size: size refers to the numbers of Direct and Support staff involved in the project. • Turnover. • Software Development Environment (SDE). The SDE is the combination of tools, techniques, • and administration used during the development.

48. Productivity • Management characteristic • It describes “how” the project is managed. Management characteristic information is recorded in four main categories: • User Participation. Record the level of participation by the user or their representative on the project. It should be considered during the project characterization. • Stability of Product Requirement. It characterize the extend that the requirement remained constant throughout development. It should be measured by the Requirements Change Distribution • Constraining Management Factors.It specify the management or administrative factors that limited the project, e.g. fixed cost, fixed staff size, fixed functionality, fixed quality and reliability, fixed schedule, limited accessibility to development system, limited accessibility to target system. • Not Directly Productive ActivitiesThese are some activities, that although necessary, do not produce a measurable and tangible artifact, like Project Management and Configuration Management. It is possible to collect the values of the effort spent for these activities for each project and these data will be considered, at the end of the project, for the productivity evaluation.

49. Productivity • Product characteristic: • Criticality: this can be: • Timing Critical: some products that must work in environments where the real-time behavior, user response, or throughput is critical. • Memory Critical: products that must be fit into a limited amount of memory. • Quality/Reliability Critical: some products that must meet very stringent quality or reliability criteria. • Degree of Innovation: the technological risk of the project. • Complexity: there are three types of complexity: • data, • programming, • organizational (the last refers to the difficulty in coordinating and communicating with all parties on the project teams). • Reuse: Some project can use already existing components (with reuse activity) or can develop • in order to produce reusable components (for reuse activity).

50. Effort • Number of persons per unit time (most common measure for planning and controlling the project progress) • Related to • work performed during each planned activity; • work expended in not directly productive activities (Project management, Configuration Management, etc.) • The collection of effort data expended on the various activities involves the gathering of the effort spent by each person on those activities, by means of proper forms • Tracking the effort data and comparing them against a reference effort distribution provides a good means for the in-process assessment of the project trend. • Effort data are also one of the input data used for the calculation of the Productivity