CSE1720 – Metrics

1. CSE1720 – Metrics Lecture 9 Measurement and Management of Performance

2. Metrics • Metrics are now heavily used for measurement and management of performance against objectives • It is sometimes called ‘Management Science’ and relies heavily on mathematical techniques

3. Metrics • As with many software packages, the user needs to have an understanding of their purpose, their validity and their relevance to a particular environment or application

4. Metrics • A metric is a measurement • In IT, ‘metrics’ are measurements of inputs, outputs and processes • The processes are widely varied – research, marketing, manufacturing, finance, ?? • Can Accounting be metricised ? • Can metrics be established for a ‘system’ ?

5. Metrics • How is the process started ? • A benchmark is established – generally this is arbitrary (or a starting point) • Experts, consultants, experienced subject planners, users, controllers, auditors (for instance) • This provides a basis for actual measurements and statistics to refine (or alter) the benchmark process – and of course its outputs

6. Metrics • Metrics are selected to provide • The most amount of information • But using the least amount of resources • To report the results in the most effective and presentable manner

7. Metrics • One definition of a process is ‘ a series of actions or operations leading to an end product’ • Another definition is an action or a continuous and regular action or succession of actions or operations occurring or performed in a definite manner • Inputs and Outputs are also ‘part of the process’.

8. Metrics • Most processes exchange inputs and outputs • There is an ‘interface’ between Inputs and Outputs • There is an Interface between Accounting and Finance • There are interfaces between Customers and a Company • Between a regulatory Body and a Company (ICCC ? Or the Department of Fair Trading ….

9. Metrics • Processes are ‘united’ or enabled by ‘Workflow’ • This is a series of tasks which share or a connected by dependencies • A workflow requires resources • A task normally has a number of of operations which the resources perform on components and the result is a product (as are your assignments).

10. Metrics • Now for some daylight :- • Each operation • Task Input and Output • Interface • Dependency • Each resource • are capable of measurement • And these measurements are the basic components for metrics which will provide ‘intelligence’ about organisation (or enterprise) performance

11. Metrics • Don’t get the wrong idea – it would be both difficult and cost ineffective to measure every component of every operation • And, is ‘every’ component of every operation ‘measurable ?’ What do you think ? • A ‘measure’ must be quantifiable

12. Metrics • How are measures described ? • They can be described using their ‘units’ • Distances can be in millimetres, centimetres, kilometres • Mass can be described in milligrams, grams, kilograms • Speed can be in mm/sec, metres/sec, kilometres/hour – and don’t forget the various speed restrictions when you are driving – you could become part of a ‘metric’.

13. Metrics • The unit of measure defines the a measure, not the entity being measured • Single valued metrics are of very little use – there is insufficient ‘data’ to develop a pattern For instance Earth to Mars flights (and possibly Mars to Earth flights) probably will not be frequent enough this decade to provide reliable metrics about anything.

14. Metrics • Measures are useful when they become part of a segment taken from a statistical sample population – such as genes, viruses, cells, neurons for instance • When we use the word ‘normal’ we are indicating that there is some time series analysis and a comparison to a basis for ‘measurement’.

15. Metrics • Victoria’s water storages are not ‘normal’ compared to storages of say 35 to 40 years ago – or are they ? • Are statistics available ? • Have conditions changed ? - population, uses ?

16. Metrics • You ‘automatically’ use ‘types’ – currency, date, character, number and so on are data types (as in Database) • Measures are also ‘typed’ • Some of these ‘types’ are • Binary • Statistical • Average • Ratio

17. Metrics • The Average age of students who enrol in Higher Degrees would be an ‘Average’ type • The number of vehicles involved in accidents (or incidents) related to the age of the vehicle, or the driver, or the make of vehicle would provide us with ratio of one measure (number of incidents) to another another measure (make) is another example of metric type • Any suggestions of other ‘types’ ? ?

18. Metrics • Another type would be Productivity • Which could be expressed as Resource Usage/number of units produced as in the case of Building Investors where the metric is the Income (in $A) per square metre pf leased space and the leased space is shown as a metric of the total available leased space / actual leased space in square metres.

19. Metrics • Metrics can be either ‘simple’ or ‘compound’ types • Simple = singular and direct – they stand alone and are not associated with other metrics and measure single or simple aspects of the area of interest • A case here would be time for an organisation to fully process a user application (rental/lease of a residence), issue of a fine as in PERIN.

20. Metrics • The entity here would be ‘processing the application’ and is measured individually for the attribute ‘time’ • There is no other interference or complexity from other ‘applications’ • Averaged over time gives the performance standard for the application processor, and then variances from the ‘standard’ can be calculated

21. Metrics • It is a ‘direct’ metric • It is not derived from another or other metrics • It is not evaluated relative to the performance of other like ‘application processors’

22. Metrics • Which leads to  Compound Metrics • These are derived (also called indirect) • They may be composite • They may be hierarchical • And overlap is possible – but this is not necessarily a good policy, and cases of overlapping metrics need to be detected and eliminated or combined

23. Metrics • It is a good rule to convert different/overlapping metrics to a common unit of measure – • Incidence of occurrences over time • Frequency • Time groupings • Money – costs or income (profits ?) • Typical units of measure are $s/hour, $ per unit produced, units per $ (or per $1,000 dollars) • Units/hour is another unit of measure

24. Metrics • Compound metrics are complex • Clear understanding and careful construction are essential to the validity and integrity of the metric • They need to be carefully built – and they must be clear and relevant

25. Metrics • Compound Metrics can be • Weighted and Composite Averages • Statistical Analyses • Layered Metrics • Thresholds and Triggers • Any others that you know of ?

26. Metrics • Weighted and Composite Averages • But weighted averages may not be accurately representative • They may be skewed • They may produce totally invalid results • If the ‘weights’ are incorrectly based, then it is unlikely that the results will be correct

27. Metrics • A Composite Average is the mean of a group of averages • They are generally assigned weights • If these is applied, the weight of the composite average can be fed back to the components which make up the components of the composite average as a percentage of the total of the component average values • This can be further investigated forsensitivity

28. Metrics • Statistical Analysis • Regression analyses • Forecasting • Correlation analyses • Variance Analyses • Co-Variance analyses • Knowledge Workers need to be familiar with these techniques

29. Metrics • Layered Metrics • Consolidation Metrics • Each metric component has a hierarchical relationship with other members of the set • In an Analytical Hierarchical Process, Subordinate (lower level) metrics can influence Superordinate (higher level) metrics

30. Metrics • Thresholds and Triggers • This allows for action to be taken on ‘out of range’ value or values. It can also initiate action on a value or values within a given boundary set. • Manufacturing is often associated with Statistical Process Control metrics. • High volume, long running processes result in large number of identical units (motor vehicles, parts, milk, medicine …..)

31. Metrics • Statistical data can be either • Periodical ( every 15, 30, 60 minutes) • Quantitative ( each given number of units processed 100, 2000 ……) • Details are plotted or charted • Upper Control Limits • Lower Control Limits • Target value • Evaluation of performance ( + or -) then triggers ‘corrective action’

32. Metrics • Alternatively, ‘trends’ - a number of increasingly negative, or increasingly positive, measurements could trigger a corrective action

33. Metrics • You should now notice a subtle change in emphasis • We will now be discussing • The relevance of a particular metric • The validity aspect • Which metrics suit which ‘businesses’ • let’s move on

34. Metrics • Any discussion on relevance and validity must be based on what effect the metric will have on the ‘business’ • You have heard of the phrase ‘at the end of the day ..’ and ‘the bottom line is ……’. • Metrics must be chosen so that their effect ‘at the end of the day’ are both valid and relevant

35. Metrics • Business goals are a statement of the business objectives • The Business goals of Monash University are : • Excellence in education • Excellence in research and scholarship • Excellence in management • Innovation and Creativity From a presentation by Vice Chancellor R Larkins April 2004

36. Metrics • Diversity • International focus • Fairness • Engagement • Integrity • Self reliance • Education

37. Metrics • And there must be a method of measurement so that performance can be evaluated • A metric may stand alone - or it may be a member of a system of metrics - you saw these mentioned in ‘compound metrics’ which report across function or processes • And every metric system and part or component, must support and be associated with measurable, clear business objectives or goals.

38. Metrics • If a metric supports the goals of an organisation, then it is relevant. • If not, it is ‘irrelevant’ and is useless or misleading • What if the goals of an organisation change ? Could they ? Could a bank for instance change its customer base and address ‘corporate’ customers only ?

39. Metrics When would the metrics alter ? How would up to date metrics be devised ? How long after the ‘change’ would ‘new’ metrics become active ? How would these relate to the superceded metrics ?

40. Metrics • Validity of a metric can be difficult • to determine • to develop • to maintain • to stay focussed • If any part of a metric becomes invalid, and this in not noticed and corrected, then the effect on a business can be fatal - do you remember the Pan-Am airline of some years ago ? • And which is now defunct ? • What happened ?

41. Metrics • Metrics in many ways are very similar to System Development and Implementation • They need to be given • Senior/Executive Management Support • Senior/Executive Management Commitment • Senior/Executive Funding They need to be seen as a valuable and reliable tool of Senior/Executive Management and Senior/Executive Management must be able to use these tools confidently and competently

42. Metrics • Validating Metrics: some guidelines • 1. The metric must relate to the goals • The relationship must be valid • It must be meaningful • Its purpose must be understood • Is it one of a family - and if it is can they be consolidated • Is there a mechanism to ‘audit’ the purpose of the metric so that if or when conditions alter the metric can be examined for continued use, alteration, cancellation

43. Metrics • 2. The components of a metric must be correct and applicable • A compound metric should be ‘decomposed’ into its parts • Each part should be ‘proven’ to be a support to the metric • Is the metric limited in some of its applications. If it is, why is it used ? • What is the effect of ‘weights’ as in weighted average. Is there a procedure to assess the credibility of the results ? • Are adjustments necessary ?

44. Metrics • 3. Data Quality. This must be both ‘high quality’ and of course ‘applicable’ data • Data needs to be quality controlled - evaluated. • The metrics which produce data must also be carefully analysed - on an ongoing basis • 4. Is the result of a metric applicable to its contribution of a result ? Or to components to which the metric is a contributor ? A metric may contribute to an upward ‘result’ as well as being ‘supported’ by a lower level metric.

45. Metrics • 5. Sensitivity Analyses These are methods of measuring or calculating the effect of input data to a metric - does it have a linear or exponential effect ? Can ‘weights’ of components be affected by incoming data - when or if this happens is there a ‘trigger and procedure’ to handle the situation (just as a comment, Excel’s Solver function produces a sensitivity report which is very useful as a reliability support to the ‘Solver solution’.)

46. Metrics • Quality is made up of a number of processes and steps • The ‘steps’ to Quality are often repetitive and circular • Quality applies to a total process • And of course the question is ‘ why does validation seem to be so important and necessary ?’

47. Metrics • You have probably noticed that ‘change’ is the dominant factor in Business. • Philosophically, Business must adapt to survive. Change has become a ‘constant’ Valid today - Tomorrow invalid or irrelevant

48. Metrics • Indicators of performance must reflect the ‘current state’ • This impacts on continuing relevance and validity of performance metrics • This requires continuing audit and maintenance • And in turn, any action taken as a result of a metric will be ‘probabilistically sound’. So now let’s have a deeper look into Organisational Metrics

49. Metrics Responsibility Accounting which occurred in the 1970’s created the need for Managers to have accurate and representative monitoring tools. Executives also must have a similar but higher level tools or tools (have you heard of ‘dashpots’ ?) Bottom up design of Organisational metrics (ascending levels of management) will succeed if there is a clear expression from Senior Management of those metrics which are seem to be essential to support the ‘Business Plan’

50. Metrics At this stage you could refer to the Monash Vice-Chancellor’s plan for this University In Business, it is generally recognised that there are 4 levels or layers in an organisation (another term is Enterprise) In Business, these layers are ‘Executive, Division, Department, Group’ In Monash they are Executive, Faculty, Department and School