Methods for Learning Metrics

1. Methods for Learning Metrics A workshop in conjunction with Metrics 2005 September 19, 2005 Como, Italy Organized by: Carolyn Seaman Letizia Jaccheri

2. Rough Schedule for Today 9am    Introductions, team-building, and preliminaries 10am    Break 10:30m   Presentations of position papers and discussion 12:30pm  Lunch 2pm    Teaching design exercises 3:30pm    Break 4:00pm Group photo 4:05pm Discussion 5:30pm Close

3. Introductions • Step 0: Name, institution, job title • Step 1: • Pair up • Share work details – research, teaching, management, development, consulting, etc. • Introduce your partner

4. Introductions • Step 2 • Pair up differently • Share who you are outside of work: • What’s important to you? • What do you enjoy? • What defines you? • Introduce your partner

5. Introductions • Step 3 • Pair up differently • Share what you have taught • Other than software engineering courses • Subject • Context • Audience • Introduce your partner

6. Presentations • Luigi Buglione, Alain Abran: • Improving Measurement Plans from multiple dimensions: Exercising with Balancing Multiple Dimensions - BMP • Linda M. Ott • Developing Healthy Skepticism not Disbelief - Problems in Teaching Software Metrics • Giuseppe Lami • Learning Metrics for Natural Language Requirements in an Under-Graduate Course • Letizia Jaccheri, Sandro Morasca • Understanding the role of software metrics in an Empirical Software Engineering Course for PhD students • overview of course • Maurizio Morisio, Marco Torchiano • TDE: a method and tool to teach OOP

7. Teaching Design Exercise • Split into 3 groups • Everyone does task 1 in their group • Each group presents their course design • Divide tasks 2, 3, and 4 among the three groups • Each group presents their work • Discussion

8. Teaching Design ExerciseTask 1 • Subject to be taught: How to change money in a foreign country • Design a course, including: • Objectives and overall structure • Outline of material to be presented • Exercises • Assessments

9. Task 1 (cont.) • Course should be based on collaborative exercises, e.g. • Producing a film or video as a group • Creating a piece of artwork • Collaboratively designing a poster on some topic • Deciding how something should be built (i.e. making design decisions) • … and student reflection and feedback, e.g. • Student presentations on reading material • Personal logs or journals written by students • Essays about lessons learnt • Oral examinations • Etc.

10. Task 1 Results • Group A • Objectives: • To provide the skills needed to end up with the needed currency in hand without breaking rules • Materials • Talks – money, value of money, different exchange rates, why it is a problem • Motivate importance of topic – why it will be helpful to them • Example scenario – going abroad to a conference • Converting cash and converting with credit card • Group exercises • Teams – one person on each team doesn’t know how to convert money, others teach him • Assessments • Short term – give same value of money in different currencies to different students and they have to convert it to dollars, then they have to write a log, which is evaluated against the objectives • Public exposition – so they can learn from mistakes and from other students

11. Task 1 Results • Group B • Main objective – to teach someone how to make available local currency • Half day course • Present an IDEF0 Process model for explaining the conversion process – inputs to outputs and mechanisms to do that • Topics: • What is money – different forms, cash, credit cards, etc. • Cost of living in different countries • Fees, commissions • Exchange rates • Where can we change money? • Outputs – traveler’s checks, cash, receipt • Changing money back into original currency • Teller machines, bureaus de change • Flowchart to describe different options • Exercise – Excel spreadsheets with change rates and world events related to changes • Role playing – purchase of currency • Assessment – role play, ordinal scale (0-3), strengths, areas for improvement • SMART evaluation • Feedback – oral to group; written through evaluation forms; improvements and teaching style

12. Task 1 Results • Group C • 8 sections: objectives, different ways to change, considerations, economy, safety and security, convenience, exercises, assessment • Objectives – assumed transactions are legal • Complete the transaction – end up with different currency than started with • Most economically and good exchange rate and good commission • Convenient and safe transaction • Different ways • Before the trip – bank or airport • During the trip – bank, airport, or hotel, ATM or change office • Considerations • Economy, safety, convenience • Ranked methods in decreasing order of: • Economy: Home bank, foreign bank, ATM, airport/hotel/change office • Safety and security: Bank, hotel, ATM, airport, change office • Convenience (depends on circumstances): Hotel, home bank • Exercises • Describe scenarios – circumstances of trip, situation • Students write what, how, why • Assessment • Groups of some size – critique each others’ responses and share ideas • Poster – Foreign Exchange 101

13. Teaching Design ExerciseTask 2 • Add to or modify the course you just designed to include a multi-faceted approach, including, e.g. • Psychological and emotional dimensions • Ethical concerns • Legal issues

14. Task 2 Results • Increased the number of constraints in the process model • Psychological and emotional – language barrier, style of writing (characters), may not know exchange rate, i.e. what to expect, security issues (older or not-computer-literate people), location of ATM, events you’re not aware of at an ATM • Ethical concerns – uneasy with government, not inclined to change a lot of cash, reduce expenditures if not comfortable with government • Legal constraints – constraints on amount of currency allowed in or out, types of money that can be exchanged (coins or bills), black market, merchants changing under the table

15. Teaching Design ExerciseTask 3 • Now consider the situation where you are teaching a course on changing money, but in the context of a more general course on travel • What are the learning objectives concerning changing money? • What learning objectives about travel could be illustrated by teaching how to change money? • How would your course be different?

16. Task 3 Results • Travel considerations – planning strategies (long term/prepaying vs. short term/more expensive/very flexible) • How to assess the need for money – local customs (credit cards accepted or not) • How to manage difficulties and travel emergencies – have multiple potential solutions • Travel planning illustrated by changing money • Differences in culture – orderly vs. chaotic, also affects changing money strategy • Different points of view • Urban vs. rural • Environment and government • Whether or not you need to change money – some places accept multiple currencies, esp. borders

17. Teaching Design ExerciseTask 4 • Now suppose that you are teaching your course, and some students say… “Why do we need to know this? With the Euro, we’ll never need to change money again!” • How do you teach them that the concepts are still important?

18. Task 4 Results • Teach them that the concepts are still important • Use an experience that they might have, e.g. someone has to plan where they work in the next 5 years, have to compare wages in different countries • Traveling and buying goods – examples to show them that this will help them understand what’s going on • Experiment – go to EBay and buy Italian wine in Euros – how much will it cost?

19. Discussion • Can we apply what we’ve designed for teaching money-changing skills to teaching software metrics? • Task 1 – collaborative exercises and student feedback • Task 2 – balancing multiple perspectives • Task 3 – “sneaking in” metrics • Task 4 – dealing with “disbelievers”

20. Applying Ideas to Metrics Teaching Currency Exchange: • Learning Objectives • To provide the skills needed to end up with the needed currency in hand without breaking rules • To teach someone how to make available local currency • Complete the transaction – end up with different currency than started with • Getting a good exchange rate and good commission • Convenient and safe transaction Teaching Metrics: • Learning Objectives • To provide skills needed to develop metrics without violating personnel rules and rules of measurement theory • Goal-driven measurement (GQM) – how derive measures from local goals (multiple levels) • End up with a quantity that is meaningful and that is understood – measure what you intend to measure – and that is useful, provides some benefit • Designing efficient measures • Measuring efficiently • Cost of collecting metrics • Return on investment of measurement • Not accepting counterfeit information – knowing when the data is bogus • Audience of data – how to use, who should use it, security of data

21. Applying Ideas to Metrics Teaching Currency Exchange: • Material covered: • Why it is important • Why it’s a problem • Basic concepts - money, value of money, different exchange rates, • Different ways of converting • Before and after trip • Bank, ATM, bureau de change, airport/hotel • Different forms of money – cash, credit cards, travelers checks • Process • IDEF0 process model • Flowchart • Inputs and outputs • Cost of living in different countries • Fees, commissions • Exchange rates • Changing money back into original currency • Economy, safety, convenience Teaching Metrics: • Material covered: • Motivation • Measurement foundations and theory • Operational definitions of metrics • Different analysis methods • Process vs. product vs. personnel metrics • Dangers in converting measurement units; different scales for effort, e.g. – needed for benchmarking • Different modes of data collection • Different measurement purposes; application of specific metrics • Different measurement scales – ordinal, interval, etc. • Measurement process • Comparing different processes • Variety of programmer salaries/costs • How measures work in different types of organizations • Use of normalized metrics • Costs of data collection/analysis • Auditing • Using measurement results for improvement • Showing progress through measurement results • Tie the results to process, not to people • Cost/benefit analysis of measurement • Evaluating/assessing intangibles

22. Applying Ideas to Metrics Teaching Currency Exchange: • Material covered (cont.): • Psychological and emotional – language barrier, style of writing (characters), may not know exchange rate, i.e. what to expect, security issues (older or not-computer-literate people), location of ATM, events you’re not aware of at an ATM • Ethical concerns – uneasy with government, not inclined to change a lot of cash, reduce expenditures if not comfortable with government • Legal constraints – constraints on amount of currency allowed in or out, types of money that can be exchanged (coins or bills), black market, merchants changing under the table • Strategy – for both travel and money changing • Travel planning illustrated by changing money • Differences in culture – orderly vs. chaotic, also affects changing money strategy • Urban vs. rural • Environment and government • Whether or not you need to change money – some places accept multiple currencies, esp. borders Teaching Metrics: • Material covered: • Culture of quality • Measurement textbooks translated into different languages • Legal obligations of measurement • Acceptance of measurement programs by developers • Paranoia • Personal and professional vulnerability • Inappropriate uses of measurement • Distinguishing randomness from trends • Using measurement for building SLA’s and other types of contracts • Third party assessment – ethics and legalities • Subversion of metrics by changing behavior • Using different measures at different stages of project and converting between them • Influence of different programming languages

23. Applying Ideas to Metrics Teaching Currency Exchange: • Exercises: • one person on each team doesn’t know how to convert money, others teach him/her • Excel spreadsheets with change rates and world events related to changes • Role playing – purchase of currency • Oral feedback to group; written feedback through evaluation forms; improvements and teaching style • Describe scenarios – circumstances of trip, situation; students write what, how, why • Poster – Foreign Exchange 101 • Comparing wages in different countries • Examples to show them how changing money will help them understand larger issues of economics and world events • Experiment – go to EBay and buy Italian wine in Euros – how much will it cost? Teaching Metrics: • Exercises: • Group members teach one member about metrics; students create pedagogical materials • Project manager version of Monopoly • Process control charts – show real examples; relate to real events, e.g. vacations, reorgs • Refactoring exercise – take measurements, refactor according to metrics, re-measure • Evaluating the validity of metrics in a published study • Role playing manager and developer • Students look at own company’s data and apply estimation models and relative errors with actuals • Scenarios – discuss how expected values would be adjusted in that situation • Give a real scenario and set of measurement data, students make inferences and predictions about project • Differences between metrics on different types of software (e.g. OS, embedded, etc.) • Compare functionally equivalent software using a set of metrics to see differences • Measurement profiles of different teams of developers – what can you get with the same amount of XXX from different teams? • Design a metrics that demonstrates your value as a software engineer • Look at history of software industry through measurement snapshots • Outsourcing exercise

24. Applying Ideas to Metrics Teaching Currency Exchange: • Assessments: • Short term – give same value of money in different currencies to different students and they have to convert it to dollars, then they have to write a log, which is evaluated against the objectives • Public exposition – so they can learn from mistakes and from other students • role play, ordinal scale (0-3), strengths, areas for improvement • SMART evaluation • Groups of some size – critique each others’ written responses to set of questions and share ideas Teaching Metrics: • Assessments: • Compare different students’ designs for the same measurement problem • Compare different students’ process assessments given same data and assessment process • Look at different metrics and how they relate to the underlying value of the software and how the metrics could be combined properly • Given a scenario, which metrics are most suitable and why? • Manager/developer role play – “why do I have to collect this?” • Role playing with appraisals – making sense of information given by a project team