Garry Law Law Associates Ltd Consulting Engineers PO Box 87311 Meadowbank Auckland

1. Using Measurements of Contractor Performance to make Informed Decisions Regarding Services6th Annual Strategic Facilities Management ConferenceConferenz, Auckland March 23, 2009 Garry Law Law Associates Ltd Consulting Engineers PO Box 87311 Meadowbank Auckland – 09 520 2152 - 0275 665764

2. Road Map Using measurements of contractor performance to make informed decisions regarding services rendered. Quantifiably measuring contractor performance is not straight forward; yet doing so yields authoritative data upon which to base decisions on regarding your contractor’s performance.  This session will reveal how best to link KPIs,  Service Level Agreement and contracts to quantifiably determine your contractor’s performance. • Extracting metrics from their performance • Best processes for interlinking  KPIs,  SLAs and contracts • Examples and guidelines for using the metrics yielded to make decisions regarding contractors Garry Law: Performance Measurement

3. Why Measure? • Truism – What you can’t measure you can’t manage • But – not everything is measurable • Relationships are hard to quantify: Tolstoy, Anna Karenina: “Happy families are all alike. Unhappy ones are each miserable in their own way" • Don’t expect the sum of a series of measures to capture the whole relationship Garry Law: Performance Measurement

4. Relationships • Working on the relationship is not just about reviewing the performance metrics • They can be a good place to start – they should be an important part of a periodic review meeting – but they are not the whole part. • Relationships are two way – not one scoring the other • A breech of duty / trust / confidence is better directly addressed than taken into some subjective performance score Garry Law: Performance Measurement

5. Why Use KPIs? • The objective in service contracts is not just cost management • Want reliability / quality / service / information • On one-off work paid on time and materials - Want productivity • Want the service aligned to your broad needs • Multiple objectives are the norm • May link performance to rewards Garry Law: Performance Measurement

6. Setting KPIs • Need to start with your organisation’s objectives (Key Results Areas) • Only some of these will involve the contract • You will have some organisational KPIs. Some will measure broad things including the relevant service. • To get organisational objectives into a useful form, cascade the objectives to the relevant level and set a KPI on that. • If the contractor can’t control most of what influences them they can’t be used. Garry Law: Performance Measurement

7. Cascading Objectives The objectives are hierarchically linked - the KPIs relate to objectives not each other Garry Law: Performance Measurement

8. Sorts of KPIs • “Milestone” Achieve something by a date. e.g. Populate the Computerised Maintenance Management System by July 30th 2011 • “Measure”Meet a target onadefined measurement. e.g. Measure: Rework rate on class 3 tasks Target: < 5% A service contract will most likely have both. Garry Law: Performance Measurement

9. Good KPI Characteristics • Output oriented • Measured for a long time • Have external benchmarks to set good performance • Contractor has primary influence over achievement • Close to real time (contractor can act to correct) • Contractor can measure (but with auditability) • Balanced set across the scope of the contract • Limited natural variation • Not too many! Garry Law: Performance Measurement

10. Output oriented • The client’s need’s are outputs not inputs • The contractor’s inputs do not guarantee the outputs will be achieved • Leave the “how” to the contractor – that’s their job not yours Garry Law: Performance Measurement

11. Measured for a long time • - If Yes - Is it still relevant? –or is it an ingrained habit? • - If No – its not fatal • Look at what other people have been measuring (they may have benchmarks as well) • If its entirely new – do a slow start – don’t make it central / vital / all important until you have a history of knowing how to measure – knowing what makes it better or worse and know what a “good” score is. Garry Law: Performance Measurement

12. Have external benchmarks to set good performance • Great if you have these • But the compromise will often be you have to measure them in exactly the same way that others do (sample size / frequency etc.) • The price of sharing BM information may be revealing your own performance (not always a bad thing, but don’t do it unthinkingly) Garry Law: Performance Measurement

13. Close to real time (so the contractor can act to correct) • If you undertake surveys on a long period interval the contractor will be in the dark as to their performance for much of the time. • If your results have some sort of delay from when they were collected the contractor will be likely not to give them much credence Garry Law: Performance Measurement

14. Contractor can measure (but with auditability) If the client does all the measurement it can lead to problems: • Master / slave relationships / • “Gotcha” approach from client • Contractor duplicating measurements in their own defence and disputing the client’s measurements Garry Law: Performance Measurement

15. Limited Natural Variation • Common Causes - variations from the contractor’s systems - manageable by them) • Special Causes - variations external to the contractor’s systems (but will include variations caused by the client) (W E Deming on Quality) Using alliance methods may allow client side issues to be worked on as well. Garry Law: Performance Measurement

16. Balanced • Akin to the ‘balanced score card’ within an undertaking • Some examples: Garry Law: Performance Measurement

17. Roading Operations and Maintenance Contract (Extra work and extension reward) 12 KPIs (Monthly to Annual) Groups: • Communications and Customer Services and Emergency Response (20%) • Health, Safety, Quality & Environmental (20%) • Operations, Physical Works (40%) • Technical Support Services (20%) Hurdle scores for extra work and extensions Garry Law: Performance Measurement

18. Water/ Wastewater / Stormwater Operations and Maintenance Contract (Extra work and extension reward) 15 KPIs (Monthly to Annual) Groups: • Communications, Customer Care, Relationship & Emergency Management (20%) • Health & Safety, Quality, Environmental (20%) • Operations, Physical Works (40% ,) • Reporting, Management of Assets, Innovation (20%) Hurdle scores for extra work and extensions Garry Law: Performance Measurement

19. Wastewater Treatment Plant Maintenance Contract (Sum at risk penalty) 9 KPIs (Monthly to annual) Groups: • Quality 70% • Reporting 20% • Health and Safety 10% Sum paid proportional to scores with range: awful / slight stretch Garry Law: Performance Measurement

20. Alliance Design Build Contract, Wastewater Treatment and Recycling Plant Construction (Quality pool reward / penalty) 12 KPIs (Monthly to whole of term) Groups: • Acceptance / commissioning 25% • Environmental compliance 15% • Operating costs 30% • Legacy 30% Sum paid + proportional to scores with range: awful / BAU / stretch Garry Law: Performance Measurement

21. Managing Contract, Water / Wastewater Reticulation Design / Construction (Quality pool reward / penalty) 12 KPIs (Monthly to annual) Groups: • Environmental outcomes 30% • Stakeholder management 30% • Legacy / quality 40% Sum paid + proportional to scores with range: awful / BAU / stretch Garry Law: Performance Measurement

22. Mathematically: Mathematically there are different sorts of measurement scales: Non- Parametric: • Nominal scales: milestones, pass/fail, yes/no, on/off, true/false, counts by classes • Ordinal scales: ranks, counts by ranks • Interval scales: defined units but no natural zero, differences are real but ratios not Parametric: • Ratio scales: defined units with a natural zero, ratios are real and unit-independent Engineers tend to be happiest with the last – but the others are perfectly respectable (and have their own branch of statistics) Garry Law: Performance Measurement

23. Defining KPIs • Units • Time window looked at • What counts • Who measures • When reported • Exceptions management • Who can correct errors A methodology datasheet per KPI is good practice Garry Law: Performance Measurement

24. Linking KPIs to Rewards / Penalties - 1 • Termination • Extra work within the contract term • Extending the contract term • Renewing the contract • Performance payment incentives / disincentives Garry Law: Performance Measurement

25. Linking KPIs to Rewards / Penalties - 2 • Use all the KPIs? – Probably not • Extra work within the contract term? – may not be an incentive if there are better opportunities elsewhere • Extending the contract term? – ditto, May be a distant reward at the beginning, in the final term extension the incentive is lost • Renewing? – Can only be one factor - a distant reward at the beginning Garry Law: Performance Measurement

26. Performance Incentive Reward is: Fixed fee for management / overheads Fixed fee for specified scope and / or Variable fee (time and materials) for unpredictable items “At risk” element variable on performance

27. KPIs linked to Payment: Sum at Risk Method • The at-risk element may be a fixed sum or a percentage of the base payment, or the profit revealed • If the client sets this too large the contractor will manage risk by bidding up the base payment • The actual payment is the “at risk” element factored by an overall performance measure • The overall performance measure is made up of a “basket” of performances on KPIs • Full reward is at a “Stretch Target” level for the KPIs. • Zero reward is an abysmal performance level Garry Law: Performance Measurement

28. KPIs linked to Payment: Quality Pool Method • The at-risk element is a fixed sum set by the client • The actual payment is the pool factored by an overall performance measure, additional if the performance score is favourable, a deduction if unfavourable • The overall performance measure is made up of a “basket” of performances on KPIs • “No payment” level is performance at a “Business as Usual” level • Full positive reward is at a “Stretch Target” level for the KPIs. • Full negative penalty is an abysmal performance level Garry Law: Performance Measurement

29. KPIs linked to Payment • Why offer more? (i.e. the quality pool method) • In a competitive environment it in fact makes little difference – the contractors will tailor their bid around the profit they are targeting, where they think they can perform on the KPIs and price accordingly Garry Law: Performance Measurement

30. KPIs within the overall performance score • Can use as “all – or – nothing” - appropriate for measures where failure has an external risk – environmental prosecution or such. Must be achievable. Don’t use this on matters which may be cause for contract termination • Can be proportional. Set the upper and lower limits and pro rata in between • With a proportional linking review if you actually want better than ‘business as usual’ for a particular KPI – don’t do it unless it has value to you Garry Law: Performance Measurement

31. Opportunities with Targets • Set them once measurement has run for a bit (on some - not too many) • Step them up – if way below a benchmark standard, get there progressively rather than pretend you will do it overnight. • Ratchet them – a new performance record becomes the new target (don’t do if there is much variation beyond the control of the Contractor i.e. Deming's special causes dominate ) Garry Law: Performance Measurement

32. Risks with KPIs Garry Law: Performance Measurement

33. END Law Associates Ltd Consulting Engineers PO Box 87311 Meadowbank Auckland – 09 520 2152 - 0275 665764 Garry Law: Performance Measurement