Software Metrics in Motorola by Daskalantonakis

1. Software Metrics in Motorolaby Daskalantonakis • Software Metric: • a method of quantifying the “extent” to which a software i) process, ii) product, or iii) project possesses a certain attribute, including: • formula for computing the metric value • chart for presenting the metric value • guideline to use and interpret the the metric values, (in context) (what metric scale is definition --- nominal, ordinal, interval or ratio?) • Software Metric must be defined with intended goal and the attributes of interest (Basili’s G-Q-M). • Implementation of software metrics is complex and involves multiple dimensions (discussed later)

2. Some Pre-requisites (tools & processes) • Tools to help collect and analyze data should exist • cost accounting system • software configuration management system • problem reporting-corrective action recording system • Processes are documented and really in use (executed) for those processes that are associated with the metric ---- remember SEI ‘s process improvement cycle where process must be in use before you can control and improve? ---

3. (5) Multi-dimensional Views of Metrics • 1) Usefulness/Utilization (for a metric to be useful ---it must be): • precisely defined and easy to understand • objective • cost effective (value of information > cost of getting it) • informative • 2) Types of Metric • process • product • project • 3) Metric audience and users needs to be considered • software users • senior management • software project managers • Software engineers • software process engineers, quality assurance engineers, management staff

4. (5) Multi-dimensional Views of Metrics • 4) Must Address Metric Users’ Needs: • gain metric consensus/acceptance • provide training and consulting support • provide tools support for data gathering, analysis, etc. • 5) Must Account for Levels (organizational) of Measurement • company wide • divisional or product group • project • sub-component • individual unit

5. Software Metric Initiatives in Motorola • Better understand the software development process ---- to improve --- not just to collect information • productivity • quality • cycle time • Two areas of focus for the metric initiatives • a) process improvement • b) project control • A company-wide Metric Working Group established to • Define metrics • Set up processes to deploy the metrics Note: the goal is NOT just to measure, but to “improve” through measurement, analysis and feedback --- a bit like the goals of SEI

6. Metrics for Process Improvement • With management support, a Quality Policy for Software Development(QPSD) establish 8 areas of focus for measurement: • Delivered defects • Total effectiveness through the process • Estimation accuracy • Adherence to schedule • Number of open customer problems • Time that problem remain “open” • Cost of non-conformance • Software reliability • 7 “Goals” for measurement: • 1) improve project planning • 2) increase defect containment • 3) increase software reliability • 4) decrease software defect density • 5) improve customer service • 6) reduce cost of non-conformance (to requirements) • 7) increase software productivity debated these 8 focus areas for a year then defined

7. Specific Metric Definitions using G-Q-M approach • Goal :Improve Project Planning • Question:How accurate is schedule estimation? • Metric: Schedule Estimation Accuracy = (actual proj. duration) / (estimated proj. dur.) • Question: How accurate is effort estimation? • Metric : Effort Estimation Accuracy = (actual project effort)/(estimated proj. effort) • Increase Defect Containment • Total pre-release Defect Containment Effectiveness • Defect Containment Effectiveness for each Phase • Increase Software Reliability • Failure Rate (in terms of execution time) • Decrease Defect Density (all normalized with “size” of product) • In-Process Faults (errors and defects) • In-Process Defects • Total Released Defects • Total Released Defects caused by the delta incremental release • Total Customer Found defects • Customer Found Defect caused by the delta incremental release

8. Specific Metric Definitions by G-Q-M (cont.) 5. Improve Customer Service (using month as the interval) • Total New Problems (post release ones) Opened in a month • Total Problems that are still Open at end of month • Mean Age of Openness of Problems that are still Open at end of month • Mean Age of Openness of Problems that are Closed at end of month 6. Reduce Cost of Non-Conformance • Cost (in dollars) spent in fixing post-release problems per month 7. Increase Software Productivity • Total Software Productivity in terms of software size per effort unit • Software Productivity (Incremental) These metrics were shown in “10-up” management reporting charts

9. Metrics for Controlling Current (on-going) Project • In-process control: ability to make decisions regarding the current project based on current or projected “status” • much of the data collected for the metrics used for earlier process improvement can also be used for current in-progress project & additionally others include: • Life cycle phase and schedule tracking by date • Cost value tracking: estimated cost, budgeted, actual, earned value where earned value = (total budgeted cost of the completed activities)/ (total budgeted cost) • Requirements changes over time tracking • Design progress tracking in terms of number of requirements designed • Fault “type” tracking to provide feedback for testing and support • Estimated remaining defects (using Rayleigh curve) • Review/inspection effectiveness tracking • problem severity/priority of both closed and open problems

10. Software Metric Infrastructure at Motorola • Metrics Working Group - across company (lasted severalyears) and performed the following activities: • Clarifying metric definitions, documentation, etc. • Provide training and consulting • Guide in specifying measurement automation and associated tools • Provide support for analysis of data and recommendations making • Championing some process (e.g. Defect Prevention Process) • Guide the actual implementation of software metrics • Aid in assessing software measurement technology (e.g. customer satisfaction measurement) • Metric Users Group • feedback and share experiences • Software Metric Symposium

11. Experiences and Lessons Learned at Motorola • Start with a small set of, possibly “imperfect,” metrics to address areas of improvement and evolve • As data are collected, start using them for in-progress project control. • Keep the metric data close to the source where the context of the information is well understood • Software project requires more than one single metric; a set of metrics is needed • Cost of software metrics initiative is about 1% of the total software resources • There is a marked change in attitudes towards quality and processes. Remember CMM Level 4&5 ? ---- this is the type of measurements that must be carried out - - well defined and deployed metrics

12. Conclusion • Motorola has been successful in implementing a company wide software metric • “minimal” resources • several areas ( cost reduction, improved estimation, ship-acceptance criteria, customer sat., etc.) benefited • Metrics can only show potential or real “problem areas” -- YOU must take the corrective actions ! Remember CMM level 5? ---- able to take actions - - - improvements through change management processes