Metrics That Matter Real Measures to Improve Software Development

1. SESSION CODE: DPR306 Metrics That Matter Real Measures to Improve Software Development Steven Borg, Principal ALM Consultant Northwest Cadence Steven.Borg@nwcadence.com

2. Every ‘best in class’ company measures software quality. There are no exceptions. If your company does not do this it is not an industry leader and there is a good chance that your software quality levels are marginal at best. • Capers Jones, Applied Software Measurement

3. What is Quality?

4. Clichés. True? • You get what you measure. • If you don’t measure it, you can’t manage it. • You cannot improve what you can’t measure. • Garbage in, garbage out. • If you don’t measure it, it’s just a hobby. • "You can't manage what you can't control, and you can't control what you don't measure. " • Tom DeMarco

5. Metrics Matter • Without metrics you can’t predict • Without metrics you can’t judge quality • Without metrics you can’t accurately estimate • Without metrics you can’t measure impacts • Without metrics you can’t improve consistently

6. Questions for you. • Is your development team more effective that they were 4 years ago? • How effective was your last dev tool purchase? • How effective was your last process improvement initiative? • What percentage of bugs get removed prior to shipping? • What is your velocity? Throughput? • Other questions?

7. What makes a metric effective? Defining metrics that matter

8. Characteristics of Effective Metrics

9. Honest? Before After Print 1.ToString() Print 2.ToString() Print 3.ToString() Print 4.ToString() Print 5.ToString() Print 6.ToString() Print 7.ToString() Print 8.ToString() Print 9.ToString() Print 10.ToString() For i=1 To 10 Do Print i.ToString() Next i 3x more productive?

10. True Metrics vs Proxy Metrics • True Metric • What you SHOULD measure • Proxy Metric • What you CAN measure

11. Focus on the true metric, not proxy metrics Tip #1

12. Determining the right metrics • Collecting metrics has a cost • At a minimum that cost includes time and effort • Tools such as Team Foundation Server can help dramatically reduce overall cost of metric collection and simplify gathering metrics • Collect only those metrics that you can directly leverage to improve your process, or validate an improvement • With powerful tools, there is a tendency to want to collect “everything” – this is a mistake

13. “The companies that wish to improve but do not measure are at the mercy of fads and chance. Progress may not be impossible, but it is certainly unlikely.” (Capers Jones, p. 67)

14. Only introduce one or two new metrics at a time Tip #2

15. Deming Cycle

16. ITIL Seven Step Improvement Process • Define what you should measure. • Define what you can measure. • Gather the data. • Who? How? When? Integrity of the data. • Process the data. • Frequency, format, system, accuracy • Analyze the data. • Relationships, trends, according to plan, targets met, corrective action • Present and use the information. • Implement the corrective action.

17. Standardize by setting triggers to alert you to slips in prior metrics Tip #3

18. Some widely used metrics with proven value A few effective metrics

19. Quality Metrics • Defect Removal Efficiency • Warning: Not always “simple” • Code Coverage • Warning: Not always “honest” • Performance Profiling • Code Coverage • Test Cases per feature • Bugs per feature (bug density metrics)

20. Sizing Metrics • Lines of Code • Warning: Not always “honest” • Function Points • Warning: Not “simple” • Effort (in hours, days, weeks, etc) • Warning: Not always “honest” or “comparable” • Story Points • Warning: Rarely “comparable”

21. Productivity Metrics • Velocity • Warning: Not always “simple” • Throughput • Lines of Code / Developer / Day (???) • Quality* • Note: Defect Removal Efficiency is highly correlated with Productivity • So highly correlated it can often act as proxy metric for productivity

22. User Metrics • User satisfaction • Warning: Not always “comparable” • Post-release bug count • Number of Help Desk calls • Warning: Not always “honest”

23. Business Metrics • Schedule Estimation Accuracy • Warning: Not always “honest” • Cost Estimation Accuracy • Warning: Not always “honest” • Scope Estimation Accuracy • Warning: Rarely “honest” • ROI Estimation Accuracy • Warning: Rarely “simple”, Not always “honest”

24. First identify your problem, THEN identify the metric Tip #4

26. Balance introduction of new metrics across different categories Tip #5

27. True Metrics vs Proxy Metrics

28. The problem today is not a deficiency in software measurement technology itself; rather, it is cultural resistance on the part of software management and staff. The resistance is due to the natural human belief that measures might be used against them. This feeling is the greatest barrier to applied software measurement. • Capers Jones, Applied Software Measurement

29. NEVER use metrics to evaluate individuals Tip #6

30. Slide Demo Process Improvement in Action A Real World Example

31. Setting the Stage • Process was ad-hoc • Delivery dates were often missed • Quality problems occurred frequently • Deployment generally problematic • Decision was made to adopt Agile

32. Goals • Provide process around software development • Through Scrum/Agile adoption • Not have failures in releases • Better quality • Track process (visibility to management) • More predictability Metrics: • Estimates vsActuals • % of iteration tasks completed / committed • Status measured by state transitions

33. Sprint 1 • No data because they were in the middle of an upgrade • There was a Sprint 0 to examine technologies for use in the project • Requirements were poorly conceived • A large part of sprint 1 was fixing the structure and content of the backlog

34. Sprint 2 (1/7/2009 – 1/23/2009)

35. Sprint 3 (1/26/2009 – 2/13/2009)

36. Sprint 4 (2/18/2009 – 3/13/2009)

37. Sprint 5 (3/16/2009 – 4/10/2009)

39. Trending Development 1 July

40. Watch your metrics closely; adjust when necessary Tip #7

41. A special case for metrics Estimation and Planning

42. “There is a perfect correlation between measurement accuracy and estimation accuracy: Companies that measure well can estimate well; companies that do not measure accurately cannot estimate accurately either.” • Capers Jones • "Optimism is an occupational hazard of programming: feedback is the treatment." • Kent Beck • “Planning and estimation are the mirror images of measurement. ” • Capers Jones

43. Demo Using historical metrics to estimate effort and timelines. Estimation with Historical Data

44. Recording actuals is hard Warning!

45. Count first, Calculate next, use Judgment last Tip #8

46. Understanding the power of tooling Team Foundation Server

47. Why ALM Tooling? • "Tools are very important element of defining a path of least resistance. If I can set up a tool so that it’s easier for a developer to do something the way that I want the developer to do it, and harder for the developer to do it some other way, then I think it’s very likely the developer is going to do it the way I want them to, because it’s easier. It’s the path of least resistance." • Steve McConnell

48. When possible, use an integrated ALM Tool to gather metrics Tip #9

49. Start using the built-in reports right away! Tip #10

50. The social aspects of collecting metrics Cultural issues with Metrics