Planning a measurement program

1. Planning a measurement program

2. What is a metrics plan? • A metrics plan must describe the who, what, where, when, how, and why of metrics. • It begins with why, laying out the goals or objectives of the project. • Next, the plan addresses what will be measured and at the same time, the plan must lay out where and when during the process the measurements will be made. • How and who address the identification of tools, techniques, and staff available for metrics collection and analysis.

3. What is a metrics plan? • The metrics plan paints a comprehensive picture of the measurement process from initial definition of need to analysis and application of results.

4. Why and what: developing goals, questions, and metrics • The GQM approach to process and metrics provides a structure from which managers and developers can derive a set of crucial project goals, plus the questions that must be answered in order to tell if the goals have been met. • Then, each question is analyzed in terms of what measurements are needed to help decide the answer to each question.

5. Why and what: developing goals, questions, and metrics • The GQM approach must be supplemented by one or more models that express the relationships among the metrics. • GQM could also be combined with process assessment frameworks like process maturity model (where an organization is rated on an ordinal scale from 1 (low) to 5 (high), based on 110 questionsabout its development process)

6. Why and what: developing goals, questions, and metrics • Table 13.1 lists the twelve questions required for a level 2 (repeatable) assessment; if any of these questions is answered "no", then the organization was automatically assessed at a level 1, regardless of the answers to the 98 other questions.

7. Why and what: developing goals, questions, and metrics • Problems existed with this approach and replaced by capability maturity model (CMM) • Each of the five capability levels is associated with a set of key process areas on which the organization should focus as part of its improvement activities, see Table 13.2.

8. Why and what: developing goals, questions, and metrics • in addition each process area is composed of a set of key practices whose presence indicates that the developer has implemented and institutionalized the process area: • Commitment to perform: policy, leadership • Ability to perform: resources, training, orientation, tools, organizational structure • Activities performed: plans, procedures, work performed, corrective action, tracking • Measurement and analysis: • Verifying implementation: reviews, audits

9. Why and what: developing goals, questions, and metrics • The capability maturity model spawned process assessment methods such as, Trillium (Canadian telecommunication) and Bootstrap (ESPRIT). • UK Ministry of Defence proposed an international standard for process assessment called SPICE (Software Process Improvement and Capability dEtermination) in 1995.

10. Why and what: developing goals, questions, and metrics • SPICE is recommended both for process improvement and capability determination. • There are two different types of practices: • Base practices (essential activities of a specific process involved in software development; functional, technical view) • customer-supplied: affect the customer directly • engineering: specify, implement, maintain • project: establish, coordinate, manage • support: enable, support performance other processes • organization: establish business goals, resource

11. Why and what: developing goals, questions, and metrics • Generic practices (institutionalize or implement a process in a general way, i.e., managerial view) • not performed • performed informally • planned and tracked • well-defined • quantitatively controlled • continuously improving

12. Why and what: developing goals, questions, and metrics • The ISO 9000 standards are used to regulate the internal quality and to assure the quality of suppliers. • Among the ISO standards the ISO 9000-3 provides guidelines for interpreting ISO 9001 in a software context. • maturity models and their assessment methods are becoming de facto standards but investigations showed unreliability of maturity measurements. Must assure that the models are appropriate.

13. Where and when: mapping measures to activities • how measurement will be integrated with normal development activities (Example 13.7)

14. How: measurement tools • After knowing what to collect and where in the process to collect them, must decide how to collect them. • it must follow 2 important principles: • it must be sufficiently simple so as to minimize disruption of normal working patterns • data must ultimately be included in a metrics database

15. Who: measurers, analysts, and audience • The metrics plan must address the people who perform each of these tasks on data: capturing, formatting, validating, analyzing, presenting, using (within the goals of the program), evaluating (effectiveness of the metrics plan).

16. Who: measurers, analysts, and audience • By taking into account as many perspectives(technical and managerial) as possible, you increase the successfulness of the measurement program.

17. Who: measurers, analysts, and audience • A metrics plan should organize roles and responsibilities according to the following guidelines: • people generating the measures should be first in viewing and analyzing the data. • Global validation and analysis of the data should be performed by other than the data provider. • Data and results should be made available only to those who need to see them.

18. Who: measurers, analysts, and audience • Usually separate measurement team working at an organizational or corporate level is formed. • useful to have people who were developers, familiar with statistics and experimental design, speak the language of the managers

19. Revising the plan • Are the goals the same? If business goals changed, metrics program's goals may need to change • Are the priorities of the goals the same? As one improvement is successful, other may be chosen • Are the questions the same? Questions relevant for the first stages of metrics program (how much to invest in metrics tools) may be replaced by questions of maturation (how much money and time are saved)

20. Revising the plan • Are the metrics the same? As the process matures, the richness of the measurements increase. some metrics may no longer be needed. • Is the maturity the same? The development process may change dramatically over time. Each change has important implications for measurement. • Is the audience the same? Many metrics programs start small, and expand to the corporation. Measurement should change to reflect the interests of the changing audience.