Lecture 21 Measurements and Metrics in Design

1. SWE 316: Software Design and Architecture Handout Lecture 21Measurements and Metrics in Design Ch 10b • Understand software measurements • Understand the different types of metrics • Understand how to measure the internal quality attributes of the software.

2. 2/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Introduction • Measurement: the process by which numbers and symbols are assigned to attributes. “You can neither predict nor control what you cannot measure”. -DeMarco “Count what is countable, measure what is measurable, and what is not measurable, make measurable“ -Galileo

3. 3/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Measurement and metrics • Software metrics are units of measurement that are used to characterize: • Software products, e.g. designs, source code, and test cases. • Software process, e.g. the activities of analysis, designing, and coding. • Software people, e.g. the efficiency of an individual tester, or the productivity of an individual designer. • Software project.

4. 4/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Software metrics • Software metrics provide a quantitative way to assess the quality of internal product attributes, thereby enabling the software engineer to assess quality before the product is built. • Metrics provide the insight necessary to create effective analysis, design, code and tests. • Product metrics can be used for general predictions or to identify anomalous components.

5. 5/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Software metrics • Although some companies have introduced measurement programs, most organisations still don’t make systematic use of software measurement.

6. 6/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Advantages of software metrics • Metrics quantitatively define success and failureand or degree of success or failure for the product, the process or the people. • Metrics identify and quantify improvement, lack of improvement or degradation in the products, the product or people. • Metrics make meaningful and useful managerial and technical decisions. • Metrics identify trends. • Metrics make meaningful estimates.

7. 7/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Disadvantages of software metrics • Metrics can be harmful if they are misused or collectedin a wrong way. • Faulty analysis (statistical) of metrics can make metricsuseless, or even harmful. • For metrics to be useful, metrics must be gathered systematically and regularly, preferably in an automated tool, which is not an easy task. • Metrics are interrelated. Usually the influence of one metric has impact on other metrics, which makes it hard to control them.

8. 8/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Disadvantages of software metrics • Collecting metrics should follow a formal procedurein order to get metrics collected correctly. This needsa lot of effort. • Metrics need to be validated in order to be used, which is again not a trivial task.

9. ? 9/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Why do we measure? • Assess the status of an ongoing project • Track potential risks • Uncover problem areas before they go “critical,” • Adjust work flow or tasks, • Evaluate the project team’s ability to control quality of software work products.

10. 10/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Who can benefit from software metrics? • Managerscan know: • How much does each process cost? • How productive is the staff? • How well is the code being developed? • Will the user be satisfied with the product? • How to improve? • Engineerscan know: • If the requirements are testable • If they have found all the faults • If the product or and the processes meet the goals • What will happen in the future

11. 11/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Measurement acceptance • No wide acceptance of software measurements: • Management personnel rarely know how to analyze, interpret or use the software metrics. • Various ways of analyzing software metrics. • Lack of measurement validation.

12. 12/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Measurement principles • The objectives of measurement should be established before data collection begins; • Each technical metric should be defined in an unambiguous manner; • Metrics should be derived based on a theory that is valid for the domain of application (e.g., metrics for design should draw upon basic design concepts and principles and attempt to provide an indication of the presence of an attribute that is deemed desirable); • Metrics should be tailored to best accommodate specific products and processes.

13. 13/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Measurement process 1- Formulation. The derivation of software measures and metrics appropriate for the representation of the software that is being considered. 2- Collection. The mechanism used to accumulate data required to derive the formulated metrics. 3- Analysis.The computation of metrics and the application of mathematical tools. 4- Interpretation. The evaluation of metrics results in an effort to gain insight into the quality of the representation. 5- Feedback. Recommendations derived from the interpretation of product metrics transmitted to the software team.

14. 14/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Metrics attributes • Simple and computable.It should be relatively easy to learn how to derive the metric, and its computation should not demand inordinate effort or time • Empiricallyandintuitivelypersuasive. The metric should satisfy the engineer’s intuitive notions about the product attribute under consideration • Consistent and objective. The metric should always yield results that are unambiguous. • Consistent in its use of units and dimensions. The mathematical computation of the metric should use measures that do not lead to bizarre combinations of unit. • Programming language independent. Metrics should be based on the analysis model, the design model, or the structure of the program itself. • An effective mechanism for quality feedback. That is, the metric should provide a software engineer with information that can lead to a higher quality end product

15. 15/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Goal-oriented software measurement • The Goal/Question/Metric Paradigm • Establish an explicit measurement goal that is specific to the process activity or product characteristic that is to be assessed • Define a set of questions that must be answered in order to achieve the goal, and • Identify well-formulatedmetrics that help to answer these questions. • Goal definition template • Analyze {the name of activity or attribute to be measured} • forthepurposeof {the overall objective of the analysis} • withrespectto {the aspect of the activity or attribute that is considered} • from the viewpoint of {the people who have an interest in the measurement} • in the context of {the environment in which the measurement takes place}.

16. 16/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Metrics assumptions • A software property can be measured. • The relationship exists between what we can measure and what we want to know. We can only measure internal attributes but are often more interested in external software attributes. • This relationship has been formalised and validated. • It may be difficult to relate what can be measured to desirable external quality attributes.

17. 17/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Metrics guidelines • Use common sense and organizational sensitivity when interpreting metrics data. • Provide regular feedback to the individuals and teams who have worked to collect measures and metrics. • Don’t use metrics to appraise individuals. • Work with practitioners and teams to set clear goals and metrics that will be used to achieve them. • Never use metrics to threaten individuals or teams. • Metrics data that indicate a problem area should not be considered “negative.” These data are merely an indicator for improvement. • Don’t obsess on a single metric to the exclusion of other important metrics.

18. 18/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Internal and external attributes External attributes Internal attributes Number of procedure parameters Maintainability Cyclomatic Complexity Reliability Program size in Lines of Code Portability Number of Error Messages Usability Length of User Manual

19. 19/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality • Data collection • A metrics program should be based on a set of product and process data. • Data should be collected immediately and, if possible, automatically. • Data accuracy • Don’t collect unnecessary data • The questions to be answered should be decided in advance and the required data identified. • Tell people why the data is being collected. • It should not be part of personnel evaluation. • Don’t rely on memory • Collect data when it is generated not after a project has finished.

20. 20/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Collection and analysis principles • Whenever possible, data collection and analysis should be automated; • Valid statistical techniques should be applied to establish relationship between internal product attributes and external quality characteristics • Interpretative guidelines and recommendations should be established for each metric • It is not always obvious what data means • Analyzing collected data is very difficult. • Professional statisticians should be consulted if available.

21. 21/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Types of metrics to measure process process metrics project metrics measurement product metrics product What do we use as a basis? • size? • function?

22. 22/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Process measurement • We measure the efficacy of a software process indirectly. • That is, we derive a set of metrics based on the outcomes that can be derived from the process. • Outcomes include • Measures of errors uncovered before release of the software • Defects delivered to and reported by end-users • Work products delivered (productivity) • Human effort expended • Calendar time expended • Schedule conformance • Other measures. • We also derive process metrics by measuring the characteristics of specific software engineering tasks.

23. 23/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Project measurement • Used to minimize the development schedule by making the adjustments necessary to avoid delays and mitigate potential problems and risks • Used to assess product quality on an ongoing basis and, when necessary, modify the technical approach to improve quality. • Every project should measure: • Inputs—measures of the resources (e.g., people, tools) required to do the work. • Outputs—measures of the deliverables or work products created during the software engineering process. • Results—measures that indicate the effectiveness of the deliverables.

24. 24/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Product metrics • A quality metric should be a predictor of product quality. • Classes of product metric: • Dynamic metrics which are collected by measurements made of a program in execution; • Static metrics which are collected by measurements made of the system representations; • Dynamic metrics help assess efficiency and reliability; static metrics help assess complexity, understandability and maintainability.

25. 25/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Dynamic and static metrics • Dynamic metrics are closely related to software quality attributes • It is relatively easy to measure the response time of a system (performance attribute) or the number of failures (reliability attribute). • Static metrics have an indirect relationship with quality attributes • You need to try and derive a relationship between these metrics and properties such as complexity, understandability and maintainability.

26. S o ft w a re m et r ic D e s cr i p t i on F a n i n / F a n - o u t F a n - i n is a m e as u r e o f th e n um b e r of f u n c ti o n s o r m e t h o d s t ha t ca l l som e o t h er f u n c ti o n o r m e t h o d ( s a y X ) . F an -o ut i s t he n um be r o f f u nc t i o n s t h a t a r e c al l ed b y f u n c ti o n X . A h ig h v al u e f o r f a n - i n m e a n s t ha t X i s ti g h t l y c o u p le d t o t h e r e st o f t he de si g n a n d c h an g e s t o X w i l l h a v e e x te ns i v e k n o ck - o n e ff ect s. A h i g h v a lu e f o r f an -o ut s u g ge sts t h at t h e o ve r a l l c om p l ex i t y o f X m a y b e h i g h b eca us e o f t h e co mp le x i t y of t h e c o n t r o l l o g i c n ee d ed to co or di n a t e t h e c al l ed c o mp o n en ts . Le n g t h of co d e T h i s i s a m ea sur e o f t h e s iz e o f a p r o g r am . G en e ra l l y , t he l a r ge r t he s ize o f t h e c o d e o f a c om p o ne n t, t h e m o r e co mp le x a n d e r r o r- p r o n e t ha t c om p o ne n t is l i k e ly t o b e . Le n g t h of c o de ha s bee n sh o w n to be o n e o f t h e m o st r e li a bl e m et r i cs f o r p r e d ic t i n g e r r o r- p r o ne n e ss i n c o m p o n e n t s . C y cl om at i c c om p l ex i t y T h i s i s a m ea sur e o f t h e co n t ro l c om p l ex i t y o f a p r o gr a m . T h i s c o n t r o l co mp le x i t y m a y b e r ela t ed t o p ro g r a m u n d er st an d a b il i t y . Le n g t h of i d en t i f ie rs T h i s i s a m ea sur e o f t h e av e r a ge l e n gt h o f di st i n ct id e n t i f ie rs i n a p r o g r am . T h e l o n g e r t h e i de n t i f i e rs , t h e mo r e l i ke l y th e y a re t o be m ea n in gf ul an d h en c e t h e mo r e u n d er st an d a b le t h e p r o gr a m . D ep t h of c o n d it i o n al T h i s i s a m ea sur e o f t h e de p t h o f ne st in g o f i f - st ate me nt s i n a p r o g r a m . De e pl y n e s te d i f n e s ti n g s t a te m e n t s a re ha r d t o u n d e r st an d a n d a r e p o te n t i al l y e r r o r- p r o n e. F o g i n d e x T h i s i s a m ea sur e o f t h e av e r a ge l e n gt h o f w o r ds a n d s e n t enc es i n d o c um en ts . T h e h i g he r t h e v al u e fo r t h e F o g i n de x , t h e m o r e d i f f i c u lt th e d o c u m e n t i s t o u n de rs ta n d . 26/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Software product metrics

27. 27/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Object-oriented metrics

28. 28/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Class-oriented metrics (by Chidamberand Kemerer) • Weighted Methods per Class (WMC): This metric is the sum of the complexities of all the local methods. • Depth of Inheritance Tree (DIT): This metric is the total number of ancestor classes in the hierarchy that can affect a given class. Classes with high DIT values are usually complex classes. • Number Of Children (NOC): This metric is the total number of immediate subclasses in the class hierarchy that inherit from a given class. • Coupling Between Objects (CBO): This metric is the total number of classes in the hierarchy to which a given class is linked and related. • Response For a Class (RFC): This metric is a measure of the set of methods that can be called by a local method in a given class. • Lack of Cohesion of Methods (LCOM): This metric is the count of pairs of the instance variable set.

29. 29/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Another class-oriented metrics • Number of Ancestor Classes (NAC): This metric captures the number of classes that have potential influence on a given class because of the inheritance relation, since with multiple inheritance there might be two values for the DIT metric. • Number of Local Methods (NLM): This metric counts the number of local methods, which are accessible outside a given class (public methods). • Class Method Complexity (CMC): This metric is the summation of the internal structure complexity of all public and private local methods. • Number of Descendent Classes (NDC): This metric is proposed as an alternative to the NOC metric, since a class that inherits from more than one class can have different values of the NOC metric. • Coupling Through Abstract data types (CTA): This metric measures the total number of classes that are used as abstract data types in the data attribute declaration of a given class. • Coupling Through Message passing (CTM): This metric measures the number of different messages sent out from a class to other classes excluding the messages sent to objects created as local objects in the local methods of the class.

30. 30/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Lack of Cohesion of Methods: example • Example: Consider a class C with three methods M1,M2and M3. • Let {I1} = { a , b . c , d , e } and {I2} = { a , b , e } and {I3} = {x,y,z}. • {I1} ∩ {I2} is nonempty, but {I1} ∩ {I3} and {I2} ∩ {I3} are null sets. • LCOM is the (number of null intersections - number of nonempty intersections), which in this case is 1. z e y d a x b c M2 M3 M1

31. 31/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Lack of Cohesion of Methods example (2) d a b c NDC={(m1, m2)} NNC ={(m1, m3) (m2, m3)} LOCM = max(2-1,0) =1 m2 m3 m1 d a b c NDC={(m1, m2) (m2, m3)} NNC ={(m1, m3)} LOCM = max(1-2,0) =0 m2 m3 m1

32. M a i n t a i n a b i l i t y M a i n t a i n a b i l i t y P o r t a b i l i t y P o r t a b i l i t y F l e x i b i l i t y F l e x i b i l i t y R e u s a b i l i t y R e u s a b i l i t y T e s t a b i l i t y T e s t a b i l i t y I n t e r o p e r a b i l i t y I n t e r o p e r a b i l i t y P R O D U C T R E V I S I O N P R O D U C T R E V I S I O N P R O D U C T T R A N S I T I O N P R O D U C T T R A N S I T I O N P R O D U C T O P E R A T I O N P R O D U C T O P E R A T I O N C o r r e c t n e s s C o r r e c t n e s s U s a b i l i t y U s a b i l i t y E f f i c i e n c y E f f i c i e n c y I n t e g r i t y R e l i a b i l i t y I n t e g r i t y R e l i a b i l i t y 32/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality McCall’s Triangle of Quality

33. 33/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Quality taxonomy operability training communicatativeness I/O volume I/O rate usability Access control integrity Access audit Storage efficiency Product operation efficiency execution efficiency traceability correctness completeness accuracy reliability error tolerance consistency simplicity maintainability conciseness Product revision testability instrumentation expandability flexibility generality Self-descriptiveness reusability modularity machine independence Product transition portability s/w system independence comms. commonality interoperability data commonality

34. 34/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Measuring quality Source: Budgen, 1994, pp60-1 • We have to turn our vague ideas about quality into measurables The Quality Concepts (abstract notions of quality properties) reliability complexity usability information flow between modules? time taken to learn how to use? Measurable Quantities (define some metrics) mean time to failure? minutes taken for some user task??? run it and count crashes per hour??? count procedure calls??? Counts taken from Design Representations (realization of the metrics)

35. 35/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Establishing a metrics program • Identify your business goals. • Identify what you want to know or learn. • Identify your subgoals. • Identify the entities and attributes related to your subgoals. • Formalize your measurement goals. • Identify quantifiable questions and the related indicators that you will use to help you achieve your measurement goals. • Identify the data elements that you will collect to construct the indicators that help answer your questions. • Define the measures to be used, and make these definitions operational. • Identify the actions that you will take to implement the measures. • Prepare a plan for implementing the measures.

36. 36/36 Introduction Advantages & Disadvantages Why do we measure? Types of Metrics Triangle of Quality Summary • Measurement: the process by which numbers and symbols are assigned to attributes. • A metricas “a quantitative measure of the degree to which a system, component, or process possesses a given attribute.” • Measurement help evaluate, assess and control quality of software work products.