Performance Metrics Chris Hamm Operations Director (acting) GSA FEDSIM

1. Performance MetricsChris HammOperations Director (acting)GSA FEDSIM

2. Quote PBSA Perspectives Contract Types Definitions Performance Metrics SLAs Implementation Details Pre-Solicitation Solicitation Pre-Award Kickoff Post-Award Measurements Exceptions & Pitfalls Topics

3. If you can’t measure it, you can’t manage it

4. Performance Based Services Acquisition FY08 – 50% of all eligible services acquisitions should be PBSA FAR Subpart 37.6 • (b) Performance-based contracts for services shall include— • (1) A performance work statement (PWS); • (2) Measurable performance standards (i.e., in terms of quality, timeliness, quantity, etc.) and the method of assessing contractor performance against performance standards; and • (3) Performance incentives where appropriate. When used, the performance incentives shall correspond to the performance standards set forth in the contract (see 16.402-2).

5. Government Perspective • Compliance – We have to do it • Funding – Less $ to do more work • Cost – Best Value for the taxpayer • Performance – Use of commercial best practices • Schedule – Congressional & Executive mandated dates But . . . PBSA is hard, especially when we are overwhelmed withwork and are starting with the old SOW

6. Contractor Perspective • Profitability – Companies are in business to make money first. • Project Success – Delivering success typically results in more follow-on business • Cash Flow – Regular payments and meeting corporate profitability is essential for publicly traded companies. Missing deliveries and failing SLAs may result in lower stock prices and replacement of personnel • Personal Services – Use of performance metrics reduces the ability for the Government to dictate the solution

7. Cost Reimburseable to Fixed Price continuum Production/Sustainment Research Development CPFF CPFF / CPAF CPIF/CPAF FPAF/FPIF/ FFP FPIF /FFP Enhancements Operations & Maintenance R&D Development / Production Higher risk, less-defined requirements Lower Risk, well-defined requirements Government assumes Contractor assumes more cost risk more cost risk As you move to the right, the more detailed and meaningful the performance measures become. . . .

8. Definitions • Performance Metric • A measure collected by the Contractor that may or may not have financial incentives & disincentives • SLA – Service Level Agreement • Part of a contract between the Government and the Contractor to provide services reliably and at a specified performance level. SLAs have financial incentives and/or disincentives FYI: There are lots of other names, like performance goals or SLOs – Service Level Objectives.

9. Performance Metrics are. . . . • A communications tool. The value of an agreement is not just in the final product; the very process of establishing metrics helps to open up communications. • A conflict-prevention tool. An agreement helps to avoid or alleviate disputes by providing a shared understanding of needs and priorities. And if conflicts do occur, they tend to be resolved more readily and with less gnashing of teeth. • A living document. This is one of its most important benefits. The agreement isn't a dead-end document consigned to the Forget Forever file. On a predetermined frequency, the parties to the review the metric to assess service adequacy and negotiate adjustments. • An objective basis for gauging service effectiveness. An metric ensures that all parties use the same criteria to evaluate service quality.

10. Definitions • Cost – The fully loaded cost of performing the service (DCAA cost accounting system). This comes into play on CPAF & CPIF, but does not apply to FFP types • Base Fee – Some Incentive & Award Fees have two components. The Base Fee is like percentage of fee that is automatically earned, while the rest is part of the award/incentive fee pool. Contractors like base fee for two reasons: (1) They automatically earn it, and (2) it helps their cash flow. Otherwise, fee isn’t earned until the end of the period • Rollover – If any fee is not earned at the end of a period, it can be lost or “rolled over”.

11. How GSA FEDSIM implements Incentives • Primarily, we have SLAs on our larger task orders • There are about 30 active projects with SLAs • Most incentives at FEDSIM are part of task orders with hybrid contract types (CPIF & CPAF) with a few notable exceptions (FFP) • SLAs tend to be implemented more on our Operations & Maintenance projects, but also exist on Development projects

12. Network Availability Server Availability Circuit Installation Move / Add / Change called a MAC Trouble Ticket Tier 1/2/3/4 Premium / Standard Call Center 1st Call Resolution Speed of Answer Abandonment Rate Customer Satisfaction Cost CPI from EVMS Budget / Savings Schedule SPI from EVMS Small Business Subcontracting Time to fill vacancies Backups COOP Invoicing Accuracy Examples of SLAs

13. Examples of Measurements • Percentages • See Chart • Hours • Indicate normal business hours for response! • Days • Always clarify business vs. calendar days! • Yes / No Most clients do not need Five 9s and could not afford it if they wanted it. The cost is prohibitive! NEW EXCEPTION - CLOUD availability

14. Philosophy • Why have incentives? • Setting performance standards, measuring performance, and understanding cost impacts is difficult. Without SLAs, all actions are inherently subjective • Incentives improve project performance only if contract administration and quality assurance functions are implemented. Otherwise, they just add cost. • SLAs should be “end to end” as much as possible. • Don’t measure email server availability if you can measure the time from when an email is sent to when it is received

15. Philosophy • SLAs should align with the client and project objective AND motivate good contractor behavior • Do not design an SLA that, when missed, motivates the contractor to give up on that measure for the rest of the period • Example: Doing an “average” over a six month period. If the contractor fails in the 1st month, there is no incentive to maintain service levels or improve over time. • Just because something can be measured doesn’t mean it should be an SLA • Example: It is easy to measure how fast someone picks up a phone, but a better measure is how long it takes to actually solve the problem.

16. Process (Pre-Award) • Draft Incentive/ Award Fee Plan and SLA Format included in Section J • Evaluation Decision: • Option 1: Specify SLAs & have all vendors bid the same • Option 2: Let vendors chose SLAs & targets • Review requirements with the client before release and see if tasks are sufficiently defined to create notional SLAs to include as recommendations • Remember that the Government’s strongest negotiation position is pre-award. • Fee Plans are unilateral before the period begins, bi-lateral during the period (if the CO allows any changes at all) Option # 2 is harder to evaluate but better post award: Better measures, more leverage after award.

17. Process (immediate Post-Award) • Ensure Contract Award includes any contractor proposed SLAs and/or changes from Negotiation • Set the percentage of award fee pool that is SLA based • Set up separate meeting from Kickoff Meeting with key stakeholders to review SLAs. • Ensure that all stakeholders agree to the SLAs • Set the “weight” of each SLA • Make sure the SLA is entirely within the control of the contractor (no mixed responsibility) • Add further definition of the SLA and all exclusions

18. Process (immediate Post-Award) • Develop the methodology of how to measure performance • Periods (monthly, quarterly, bi-annually) • Set Levels (Stretch, primary, secondary, tertiary) • Set Measurement Scale with Incentive / Disincentive • For Example: • Stretch: 100% + rollover • Primary: 100% of available fee • Secondary: 90% • Tertiary: 75% • Failure: 0 % Fee Not all levels are required. Some only have one target. It depends on the client requirements.

19. Award Fee Plan scale: Outstanding (95-100) Excellent (90-95) Good (80-90) Average (70-80) Poor (0-70) SLA Measurement Scale: Outstanding 99.99+ Excellent 99.9 – 99.99 Good 99 – 99.9 Average 98 – 99 Poor >98 Process (immediate Post-Award)

20. Process (a few months post award) • Hold a regular review of the metrics to ensure that the levels are set correctly • Create a dashboard for simplified reporting to executive management • Set up a quality assurance function to check the contractor’s self reporting metrics • Insist on access to the raw data from the system tools used to generate the SLA reports • Review all trends and focus on failures • Root cause analysis can identify process issues

21. Process (a few months post award) • Set a structure for gradual improvement of SLAs over time • For particularly difficult SLAs to negotiate, consider benchmarking them for a period without making them SLAs to obtain buy-in and set the appropriate target • Gradually increase the overall weight of SLAs in the award fee plan overtime, with the goal of eliminating subjectivity as much as possible

22. Exceptions • Scheduled Downtime • Government Intervention in Contractor Response • Other Contractor Intervention • Acts of God / Force Majure Make sure someone from the Government is aware of and approves all scheduled downtime or deviations from SLAs. No fox in the hen house! Another benefit of outsourcing your infrastructure to the “cloud” . . . No more scheduled downtime!

23. Measurement • Recommend use of Contractor owned automated tools • If not owned by the contractor and proposed at time of award, be prepared to pay for the tool • Some clients have their own tools. Make sure the client can extend the license to contractors • Use an IV&V contractor to sample the data from the contractor • Send the monthly results of the SLAs to stakeholders so there are no surprises when it comes time to pay the contractor

24. Structure & Rollover • Design your SLAs to motivate improved performance on SLAs overtime. • If one SLAs is missed during a period, put some rollover on the table to improve the performance against that measure for the future and earn back the lost fee. That will motivate the contractor to achieve a higher standard for the remaining period and result in better service to your client. • Example: If a contractor misses a measure in Period 1, put some rollover towards that measure (and that measure only) in Period 2 if they meet a stretch goal. You can build this into your award fee plan’s structure to happen automatically • Note: This is not always appropriate. Some failures need to have financial disincentives to change behavior.

25. Pitfalls • Having too many measures • Example: Too many SLAs, some of which were redundant and worth less than 1% of the pool • Example: Aligning SLAs to Government staff. At least one SLA (and sometimes more) for each of 25 tasks to appease stakeholders • Targets • Just like performance reviews, if you set a target, the contractor will plan and staff to meet that target. If the target is an average, make sure that they plan to make the average, not to always beat the target.

26. Pitfalls • Funding • If the SLAs has an incentive, that amount must be fully funded to prevent any anti-deficiency situation • This is what makes really good incentives difficult level of oversight • Oversight • Clients and Contractor may initially balk at the level of effort required to stand up a SLA management process

27. Pitfalls • Reliance on Industry Standards • Benchmarks are great, but that does not mean they are right for your client’s infrastructure. Just because Gartner says five 9s is normal for a call center doesn’t mean that your requirement needs that. • Customer Satisfaction • This one is difficult. It will take time to reach agreement on what scale to use (1-5, Y/N, etc), what survey to use, how to count the no-response pool. • At the end of the day, customer satisfaction may be the most important measure but the hardest to get right

28. Pitfalls • Tendency to award very high scores on Award fee (Government wide)

29. Questions