INFO 636 Software Engineering Process I Prof. Glenn Booker

1. INFO 636Software Engineering Process IProf. Glenn Booker Week 7 – Measurement INFO636 Week 7

2. Measurement • Measurements are critical, because the act of measuring something focuses people’s attention on it • We have already looked at several basic measures in the PSP • Now expand to a more general process for choosing measurements and how to present them INFO636 Week 7

3. Measurement • The approach presented here is a variation on the GQM approach cited in the text, called GQ(I)M • The (I) adds Indicators, which are the means used to present the measurements graphically • This approach is also used in INFO 630 INFO636 Week 7

4. Why Care About Measurement? • To quote Albert Einstein, "Not everything that counts can be counted and not everything that can be counted counts." • We are seeking to identify things • 1) which can be counted, and • 2) which count toward achieving our goals INFO636 Week 7

5. Reasons for Measurement • Measurements are required by all major process models (CMM, ISO 9000, etc.) • To Characterize or understand the current status of activities or products • To compare that understanding to our objectives, and Evaluate whether the current status is “good” or “bad” INFO636 Week 7

6. Reasons for Measurement • To Predict future performance, based on past trends • To form the basis for measuring Improvement • You won’t know if you improved, if you don’t know where you started! INFO636 Week 7

7. Where do we get Information? • Metrics are often used to support decision making (the Evaluate step on a previous slide) • Decisions should be based on quantified information • To get that information, we calculate measures from raw data called data elements • The data elements each come from a data source INFO636 Week 7

8. How do we Choose what to Measure? • A commonly used method for selecting measurements is called GQ(I)M for Goal, Question, Indicator, and Measurement • It is based on GQM work by Victor Basili (first reported circa 1988-89) • GQ(I)M was published by Robert Park et al INFO636 Week 7

9. How do we Choose what to Measure? • The GQ(I)M method uses ten steps to describe measurements systematically • The steps don’t have to be followed in the order presented; their main purpose is to help ensure that measurements have been fully thought out • You can start at the top, or the bottom, or in the middle INFO636 Week 7

10. 1. Identify Business Goals • Business goals are the big, vague, lofty desired accomplishments for an organization • Think of what you’d find bragged about in a company’s annual report • Reduce cycle time • Improve customer satisfaction INFO636 Week 7

11. 1. Identify Business Goals • Develop detailed process history • Respond to changing business environment • Reduce overhead • Improve competitive position • Increase market share • Improve product quality INFO636 Week 7

12. 2. Identify Desired Knowledge • Break down each goal into products, resources, and activities needed to meet those goals • Think of questions like: • What activities do I manage or execute? • What do I want to achieve or improve? • To do this, I will need to … INFO636 Week 7

13. 3. Identify Subgoals • Set subgoals (objectives) for each of the areas you manage • What do you want to know about the results of step 2? What kind of information is important to you? • How big/fast/expensive/complex/much time will a process/product/tool/resource take? INFO636 Week 7

14. 4. Identify Entities and Attributes • Formulate questions to identify entities (documents, work products) created by your process, and the attributes of them you are interested in (size, quality, duration, cost) • Don’t get too detailed at this point - just identify the type of information desired (what do you want to understand), and the subject of that information (what about it do you want to know) INFO636 Week 7

15. 4. Identify Entities and Attributes • Entities that you want to measure can come from any of: • A process’ inputs • The activities within a process • The process’ outputs • Then decide what characteristics of each entity are of interest INFO636 Week 7

16. 5. Define Measurement Goals • First choose the type of measurement goals • Active goals: reduce or improve something • Passive goals: identify, assess, understand • Lower maturity organizations start with passive goals, then work on active goals after some history is known INFO636 Week 7

17. 5. Define Measurement Goals • Form structured statements of the measurement goals for each attribute • This step defines a metric in the form of a sentence INFO636 Week 7

18. 5. Define Measurement Goals • The general format or syntax is: • <verb> <measure> <qualifier(s)> [objective] • where • <verb> is active or passive, as chosen earlier • <measure> is the actual measure to be collected • <qualifiers> indicate the scope or time frame of the measurements (a.k.a. independent variables), • [objective] is only given for active measurement goals – what value is desired? INFO636 Week 7

19. 5. Define Measurement Goals • Passive measurement goal examples: • “Measure the number of requirements which changed each month.” • “Identify the voluntary turnover rate per month for programmers and software engineers.” • Active measurement goal examples: • “Reduce the defect rate of developed code over time to under 20 defects/KLOC” • “Improve the percent of satisfied customers after 30 days of product use to 95% or more” INFO636 Week 7

20. 5. Define Measurement Goals • DO NOT report traits of individual people (for fear of judgment) • Unless that’s part of their job description (e.g. sales people) • Also need to balance how often measurements are made • More frequent measurement gives finer control, but excessive measurement wastes time and slows the process being measured INFO636 Week 7

21. 6. Quantify Questions and Indicators • Pose questions to address your measurement goals (quantifiable ones, if addressing active goals) • Active: “Can we resolve customer emergencies, on average, in under 24 hours?” • Passive: “How many requirements do we have at the end of the Requirements Definition phase?” INFO636 Week 7

22. 6. Quantify Questions and Indicators • Sometimes the search for a meaningful metric starts with a question, and that leads to filling out the GQ(I)M from the middle • Identify indicators to show the results effectively (see later this lecture), such as: • Pie charts • Bar graphs • Scatter plots, etc. INFO636 Week 7

23. 7. Identify Data Elements • Identify the data elements needed to prepare (calculate) each indicator • e.g. “defect rate by module each month” needs a list of defects found, with the module each came from, for the last month, and the size of each module in kSLOC INFO636 Week 7

24. 7. Identify Data Elements • Determine the source for each data element – from where do you get it? • E.g. defect data will come from our change request database • Size data will be generated by the development environment INFO636 Week 7

25. 8. Define Measures • Define exactly what you mean by each measure • Use a checklist if needed to show what is and isn’t included in its definition (such as we used for definition of LOC) • Cite the source if an unusual measure is used; or if you made it up, explain why • New measures are allowed! INFO636 Week 7

26. 8. Define Measures • Include any rules, assumptions, constraints, and environment which are part of the measure’s definition • E.g. Defect rate = (# of defects whose origin was in each module)/(# of kSLOC in that module) INFO636 Week 7

27. 9. Identify Measurement Implementation • Deciding how to implement a measurement program is generally done in three steps • Analyze what measures are currently collected (if anything) and how they’re used • Diagnose how well the current measurements meet your goals – find what’s missing? INFO636 Week 7

28. 9. Identify Measurement Implementation • Act on implementing new measurements, possibly in a phased approach based on project or organizational priorities • Implementing measurements is a cultural change, so like any other change, it’s best done in small steps • Add a few measurements now, wait six months, add a few more, etc. • Let people see benefits from the new measurements! INFO636 Week 7

29. 10. Prepare Measurement Plan • Take all of the aforementioned information and create a complete plan to identify and implement measurement for your organization or project • This is generally called a Metrics Plan or a Measurement Plan INFO636 Week 7

30. Summary of GQ(I)M Steps • Goal • Subgoal (objective; why collect this metric) • Question (answered by this metric) • Indicator (how display metric) • Measurement (the actual metric and its definition) • Data Elements (used to calculate metrics) • Source (of each data element) INFO636 Week 7

31. Indicators • Indicators are the means used to present measures, such as charts, graphs, etc. • Consider how your data will be presented - in color or B/W, live or printed • Will your audience see pristine originals, or will it be copied a zillion times? INFO636 Week 7

32. Indicators • To choose the right indicator, consider: • The Amount of Data to be presented for each interval (e.g. one measure at a time or five different survey responses at once) • The Number of Intervals to be shown, such as time units, modules of code, etc. INFO636 Week 7

33. Indicators • Different indicators are better at different Amount or Number characteristics INFO636 Week 7

34. Indicators • For most graphs: • The X-axis of a graph (the horizontal line) is the independent variable; often a <qualifier>, such as time, severity, etc. • If the X axis is Time, it should show how often measurements are made (weekly, monthly, etc). INFO636 Week 7

35. Y Dependent variable (measure) x x x x x x x x X axis Independent variable (qualifier) Indicators • The Y-axis of a graph (the vertical line) is the dependent variable; the thing you are measuring (the Measure). INFO636 Week 7

36. Pie Chart • The lowly pie chart is good for presenting a small amount of data • % of customers satisfied and not satisfied • % of defects by severity at this moment • Amount of Data: Shows a few data points (2-10) • Number of Intervals: One - it generally shows only one moment in time INFO636 Week 7

37. Sample Pie Chart INFO636 Week 7

38. Ishikawa’s Basic Tools • Developed circa 1989 for manufacturing production • Used widely in quality control • Focuses on project level concerns; is this batch good enough to accept? • Not very useful for research; has little theoretical basis INFO636 Week 7

39. 1. Check sheet • Used to gather data easily, consistently, and in a standard format • A check sheet used to help the quality of a process or product is a “checklist” • Helps to define key parts of a process • Examples include code inspection checklist, detailed test procedures INFO636 Week 7

40. 2. Pareto Diagram • Used to identify problem areas - where to fix first; what are the biggest fires to put out • Defects tend to cluster in buggy portions of code • Use Pareto to plot a column graph of the defect rate by the module it came from • Could add a line graph for the cumulative % of defects above it (not shown) • Pareto diagram must list X axis categories in descending order of value INFO636 Week 7

41. 2. Sample Pareto Diagram INFO636 Week 7

42. 3. Histogram • A bar chart is used to break down data by an ordered category (e.g. defect severity, satisfaction rating) • Can choose to put bars next to each other (clustered), or stack them. • Stack when they add to a constant (e.g. 100%), or when the total is also a useful measure (total number of defects) INFO636 Week 7

43. 3. Histogram • Can show limited time spans, e.g. a few time intervals Simple bar graph  INFO636 Week 7

44. 3. Histogram  Stacked Clustered  INFO636 Week 7

45. 3. Histogram • A true histogram can group ranges of values in the X axis, and show counts or average Y values for each group • E.g. average salary (Y axis) for employees aged 18-29, 30-39, 40-49, etc. (X axis groups) • Good for hiding individual values, and looking at larger trends in the data INFO636 Week 7

46. 4. Run Charts • Plots some measure versus time using a line graph • Often compare values to a desired or target value, especially at higher process maturity levels (CMM Level 3 and up) • Special case: The “S” curve plots (cumulative % completion of something) versus time INFO636 Week 7

47. 4. Sample Run Chart INFO636 Week 7

48. 5. Scatter Diagram • Plot two measures against each other to see if there’s a correlation between them • E.g. Defect rate per module versus module complexity, or productivity versus experience • When we say plot ‘Blah versus Ick,’ usually Blah is the Y axis, and Ick is the X axis INFO636 Week 7

49. 5. Scatter Diagram INFO636 Week 7

50. 5. Scatter Diagram • Once there’s enough data, can add curve fitted lines, and +/- variances • The scatter diagram is the basis for regression analysis • Curve fitting to the data, to help define a statistically significant connection between the two measures INFO636 Week 7