Measurement

1. Measurement COMS 309 David Weiss Spring, 2010 weiss@cs.iastate.edu

2. Problems in software development COMS 309 Weiss Spring 2010 Measurement Knowing what the customers want; knowing the requirements Predicting time to develop Predicting resources needed to develop Managing change Knowing when you’re done Designing to enable distribution of work Coordinating work Knowing what the value is (how much will I make) Tracking time, resources, quality, productivity, effectiveness, etc.

3. Solving Problems and Knowing It Requires Measurement COMS 309 Weiss Spring 2010 Measurement What should we measure? How should we develop the measures? How do we get the data for the measurement? How do we analyze the data?

4. Goal, Question, Metric: A Measurement Paradigm (1) COMS 309 Weiss Spring 2010 Measurement • Establish the goals of the project • Examples: • Keep customers happy • Generate members of a product line three times as fast as previously • Learn methods for, advantages of and difficulties of working in a team • Define questions that, when answered, establish progress towards the goal • Define measures that, when computed, answer the questions and thereby establish progress towards the goal • GQM

5. Goal, Question, Metric: A Measurement Paradigm (2) COMS 309 Weiss Spring 2010 Measurement • Define questions that, when answered, establish progress towards the goal • Examples: • Keep customers happy • How many customer complaints did we receive last month? Six months ago? • How many failures per installed product were there last month? • Generate members of a product line three times as fast as previously • How long did it take to develop the last 5 products using the old methods? • How long does it take to develop a product now? • Learn methods for, advantages of, and difficulties of working in a team • What were the primary difficulties each team had? • Which team produced the best products? The worst? What were the differences in team dynamics between the two?

6. Goal, Question, Metric: A Measurement Paradigm (3) COMS 309 Weiss Spring 2010 Measurement • Define measures that, when computed, answer the questions and thereby establish progress towards the goal • Examples: • Keep customers happy • How many customer complaints did we receive last month? Six months ago? • Measure: Customer complaints per delivered system per month • How many failures per installed product were there last month? • Measure: Failures per installed product per month as a function of time • Generate members of a product line three times as fast as previously • How long did it take to develop the last 5 products using the old methods? • Measure: Time to develop a product, as a function of time • How long does it take to develop a product now? • Measure: Time to develop a product, as a function of time • Learn methods for, advantages of, and difficulties of working in a team • What were the primary difficulties each team had? • Measure: difficulties described by teams and observed by instructors, prioritized • Which team produced the best products? The worst? What were the differences in team dynamics between the two? • Measure: Usefulness of products, as judged by instructors and outside observers • Measure: Difficulty of use of products, on a scale of 1-5, as evaluated by instructors • …

7. Predicting Software Development (50 sampled projects) Project Estimates At Gate 1 Project Estimates at Gate 2 Median 200% 150% Relative Schedule Range 100% 75% 50% Plans and Requirements Feasibility Product Design Development and Test Detailed Design Launch COMS 309 Weiss Spring 2010 Measurement

8. How Does Productivity Change Over Time? (1) Developer Churn for a Sample Project Project Moved Offshore To Inexperienced Group COMS 309 Weiss Spring 2010 Measurement

9. Interval Quality • Probability that a customer observes a failure within one, three, and six months after installation • Drawback • does not account for the proximity to launch • Significant differences are marked with *, **, and *** • Priorities changed from time-to-market to quality Release COMS 309 Weiss Spring 2010 Measurement

10. Objectives COMS 309 Weiss Spring 2010 Measurement Learn methods for, advantages of and difficulties of working in a team, similar to industry experience Understand why software development organizations establish and attempt to follow well-defined software lifecycle processes. Produce a working application Have some fun Students will form their own teams; each team will have a set of defined roles, such as systems engineer, architect, developer, tester, and project manager. Students may play more than one role in a team. All teams will start with a defined product line, including a set of documents and code, and each team will develop a new product in the product line. Each team will give a progress report midway through the semester, and each team will demonstrate its product at the end of the semester and will create a poster describing the product and the team’s learnings as a result of developing the product.

11. GQM: Data Collection COMS 309 Weiss Spring 2010 Measurement • Data collection should be • Unobtrusive: don’t add overhead to existing jobs • Valid: data can be used to answer the question(s) of interest • Verifiable: accuracy of the data can be determined and estimated (may need to sample) • Repeatable: different people collect the same data and get the same results • Types of data collection • Quantitative: objective, numerical data • Qualitative: opinion surveys, interviews • It’s never that simple

12. Data collection: an example of a qualitative approach COMS 309 Weiss Spring 2010 Measurement • Goal: When new developers join a project, train them effectively and efficiently • Questions: • What does a new developer need to learn when joining a project? • What help was available to new developers? • What resources did new developers have to reference? • How did new developers resolve problems? • Approach: • Designed interview questions, focusing on developers’ involvement: • Sampled people for interviews: 3 developers per project, focused on those who made most changes to shared files • Explained our purpose before the interview, and promised privacy of their identity • Conducted the interviews through telephone conferences, one project each time with two colleagues helping f2f

13. Interview Questionnaire, Part 1: Involvement in organization and project COMS 309 Weiss Spring 2010 Measurement a. When did you graduate from the university? b. What was your former job before you joined the company? c. Did you take any training courses when you joined the project? How long did they last? What were the contents of the training course(s)? d. What does your project do? Which part do you work on, such as developing a part of a system, fixing bugs (current engineering), and so on? e. How do you receive your assignment (supervisor, self)? Why were you assigned to work on this (technical background, your interest, availability)? f. What did you have to learn for your assignment, e.g., language, architecture, how to use version control/problem tracking, how to build the project specific development environment, how to write a test, how to understand the existing code? What is the most difficult part? g. What techniques did you use to understand the existing code? What factors influenced your understanding, such as code style, code comments, nobody to ask, etc? h. Does your project require code style/code comments? i. What help was available, such as reading documentation, asking a specified mentor, asking peers, when you came into the project? How has that changed? j. What kinds of support can you provide (questions, help with code changes, documentation, build, test, code understanding)? l. Please give some examples of particularly useful help you got from (or provided to) others. m. What’s the biggest barrier when you get/provide help? j. What kind of documents do you have, such as development document, requirements document, design document, test document? What are the most useful documents for you? k. What did you have to learn by yourself, not included in the documents, and difficult to get from any others? Please give an example.

14. Interview Questionnaire, Part 2: Factors that affect the outcome of succession COMS 309 Weiss Spring 2010 Measurement a. How many people are in your project? Who is your mentor? b. Who are the people who are the people whom you are most likely to ask questions? How many people ask you questions or need your support, how many people do you ask questions or need support from? c. How many people from your project are on your site and how many are on another site? d. What’s the most important thing you learned from the people with whom you worked? (coding style, how to build/test, interface, ..) e. How did you learn that, such as from discussions, reading his/her code,observing development habits, etc.? f. What kinds of meeting do you have (size: project, subgroup, one-on-one, type: progress, coordination, resolve a specific issue)? g. How is the meeting conducted (Is there any agenda before the meeting? Who hosts the meeting? What do you discuss and decide in the meeting?) h. How often do you have meetings? i. What helps most to be more productive in the project? (code comments, documentation, training, f2f meetings...) j. What could be improved to make you more productive? (better training, knowing who can help you, better code, ...)

15. Interview Questionnaire, Part 3: Additional Information COMS 309 Weiss Spring 2010 Measurement a. What’s most difficult for you? b. What would make you happier? c. Is there anything else we should have asked but did not?

16. Data collection: an example of a quantitative approach • Goal: enhance developers’ learning when joining projects • Questions: • What does learning better mean? • Higher productivity • How do developers learn? • Through practice and over time COMS 309 Weiss Spring 2010 Measurement

17. Measure productivity as changes • Software is created by making changes to it • A delta is a single checkin (ci/commit/edput) representing an atomic modification of a single file with the following attributes • File, Date, Developer, Comment • Other attributes that often can be derived: • Size (number of lines added,deleted) • Lead time (interval from start to completion) • Purpose (Fix/New Feature) • Approach • Use project’s repositories of change data to model (explain and predict) phenomena in software projects and to create tools that improve software productivity/quality/lead times COMS 309 Weiss Spring 2010 Measurement

18. Change Hierarchy COMS 309 Weiss Spring 2010 Measurement

19. COMS 309 Weiss Spring 2010 Measurement

20. Measure learning as proficiency at making changes • Developer productivity • number of changes per staff-month • Learning experience • tenure, i.e., the months from hiring day • practice, i.e., the changes the developer has made up to that month COMS 309 Weiss Spring 2010 Measurement

21. Developers learn through practice and over time • X: months since joining the company • Y: changes per developer that month (average over developers who made changes) A B C COMS 309 Weiss Spring 2010 Measurement

22. An example of GQM in software engineering AT&T used GQM to help determine which metrics were appropriate for assessing their inspection process(1994) Goal: Plan Questions and metrics: How much does the inspection process cost? Average effort per KLOC Percentage of re-inspections How much calendar time does the inspection process take? Average effort per KLOC Total KLOC inspected COMS 309 Weiss Spring 2010 Measurement

23. Effort and productivity in Trustie Goal: understand how project is going Questions and metrics: How much effort is spent per month/week? developers/month or week What is the output? Productivity (changes/time unit) COMS 309 Weiss Spring 2010 Measurement

24. Background Software is created by changes Changes are tracked COMS 309 Weiss Spring 2010 Measurement

25. Effort and Productivity COMS 309 Weiss Spring 2010 Measurement

26. Defect prediction Goal: Predict bug fixes, e.g, in order to plan effort Questions: What are factors associated with bug fixes? Metric: Use new feature mrs to predict bug fixes, both number and trend Assumption: A change may introduce defect(s) with some delay The new-feature changes are dictated by the business environment and are assumed as given New feature mrs in the past releases New feature mrs in the current release COMS 309 Weiss Spring 2010 Measurement

27. Obstacles to measurement Lack of focus on software measurement Low priority except in emergencies Need for immediate results (short time horizon) Lack of resources for measurement/improvement Difficulty of obtaining, understanding, and validating data Multiple stakeholders (developer/support/product management) Difficulty of comparison among projects, even earlier versions within the same project Establish infrastructure for data collection and analysis? Dashboards Automated data collection and analysis 27 COMS 309 Weiss Spring 2010 Measurement

28. "When you can measure what you are speaking about, and express it in numbers, you know something about it, but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science.” William Thomson, Lord Kelvin COMS 309 Weiss Spring 2010 Measurement

29. Summary COMS 309 Weiss Spring 2010 Measurement Need to measure to understand progress towards goals GQM is a useful measurement paradigm Qualitative and quantitative measures Unobtrusive, automated data collection Meaningful measurement is difficult Comparison among projects is difficult

30. Terminology COMS 309 Weiss Spring 2010 Measurement Goal, Question, Metric (GQM)

31. Exercise 3: Measuring Product Lines COMS 309 Weiss Spring 2010 Measurement Identify two goals of software product line engineering For each goal that you identified, propose three questions that help gauge progress towards the goal For each question that you proposed, define one measure that helps answer the question For each measure, sketch or tabulate results that you might expect to get

32. COMS 309 Weiss Spring 2010 Measurement

33. Expected Outcomes COMS 309 Weiss Spring 2010 Measurement

34. Exercise: Measuring Product Lines (2) COMS 309 Weiss Spring 2010 Measurement • For each goal you identified, • Goal 1: • Goal2: