Breakout Session # 804 Mike Cameron, Senior Associate, Booz Allen Hamilton

2. Writing Performance Criteria for Professional Services Breakout Session # 804 Mike Cameron, Senior Associate, Booz Allen Hamilton Date: 26 April 2005 Time: 11:00 AM NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

3. A Short Story* Long ago, a traveler came upon three men working beside the road. He asked the men “What are you doing?” The first man answered “I am shaping this rock.” The second man replied “I am building a wall.” The third man said “I am building a cathedral.” * Excerpted from The Articulate Executive by Granville N. Toogood, McGraw-Hill, 1997 NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

4. This Story Illustrated the Difference Between Effort and Results • The first man described an activity…shaping a rock • The second man described a process…building a wall • The third man described an outcome…building a cathedral Traditional service procurements are like the first two men—describing activities and processes. Performance-Based Service Acquisitions are like the third man—defining a resultin a way that reveals the purpose of the effort. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

5. The Burning Issue! • Legislation requires PBSA for service procurements—including professional services • Many agencies—and industry—are struggling to understand how to use PBSA for professional services • It’s not a small problem—professional services accounted for 34%--or approximately $31B--of service procurements in 2003 NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

6. PBSA Is Structured Around Two Key Documents… • The Performance Work Statement • Defines the results that are required • Describes what makes the results acceptable • Summarizes a performance measurement plan • The Quality Assurance Surveillance Plan • Defines what performance data is required by the government • Describes how performance data will be collected • Explains the process used to evaluate contractor performance Our focus is on the Performance Work Statement NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

7. Defining Performance—The Performance Work Statement • The Performance Work Statement (PWS) provides • Performance Requirements—unambiguous statements that describe specific, measurable results that must be provided • Performance Standards—simple statements that define the properties that make the results acceptable to the customer • Performance Measurements—descriptions of the methods by which the work will be compared to the standards • The PWS focuses on results and avoids describing how the work will be performed NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

8. Prevailing Guidance on PBSA Does Not Address Professional Services • Existing examples of performance work statements typically illustrate PBSA for “commodity” services • There is little, or no, guidance widely available that addresses professional service acquisitions • Without examples or guidance, many managers don’t know how to define performance criteria for professional services “You can’t really measure the kind of work we do.” “Our job is always changing.” NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

9. PBSA Does Work For Professional Services! • Professional services are fundamentally different from commodity services • Understanding those differences is the key to writing performance criteria that are meaningful and measurable NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

10. Commodity services are: A commodity in the marketplace Have adequate competition to set a market price Differentiated mainly by price Repetitive by nature Insensitive to context Professional services are: Usually highly customized Often fill a niche in the market Differentiated on factors other than price Non-repetitive Determined almost entirely by context Professional Services are Different from Commodity Services Transportation Facilities & Maintenance Grounds keeping Engineering & Technical Studies Program and Acquisition Support Strategic Planning NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

11. Commodity Services—Facilities Maintenance The results of the services—painting, repairs, and cleaning—are the same each time they are provided The primary function of the facility makes little difference to the delivered service Experiences gained from one project are directly applicable to another project Professional Services—Strategic Planning The specific form of the services will vary according to the need of the customer The result of the service will differ according to the unique situation of the customer Experiences gained from one project provide insight into other projects, but are not directly applicable Two Examples of Repetition and Context NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

12. Repetition and Context • The key differences between commodity and professional services are repetition and context • When a service is insensitive to context, each delivery of the service is approximately equivalent to all other deliveries • Repetition of a service allows frequent observation and determination of cause and effect…i.e., standards emerge from past experience Professional services are driven by context and are non-repetitive…performance criteria must be derived from other sources. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

13. The Best Starting Point…Focus On The Solution Focus is achieved by asking two questions • “What problem am I trying to solve?” • “How will I know when I’ve solved it?” Use the Performance Work Statement to provide the answer to these questions. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

14. Relating the First Question to Performance… • “What problem am I trying to solve?” • If you can answer this question, then you can identify the performance requirements… Performance requirements define the results that must be produced in order to achieve the desired solution. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

15. And On to the Second Question… • “How will I know when I’ve solved it?” • If you can answer this second question, then you can identify the performance standards… Performance standards define the quality levels that must be achieved, or the conditions that must be satisfied, to make the results acceptable. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

16. A Simple Example • Objective Statement: You have been tasked to create a customer service portal for your agency and you need decision support to select an appropriate technology solution. • We must translate this objective into a set of performance criteria • The performance analysis requires • Deciding what results are necessary to accomplish your objective • Identifying the conditions that make the results acceptable NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

17. A Simple Example—Analysis of the Problem • Question #1: “What problem(s) am I trying to solve?” • Answer: The measurable, specific results that must be provided in order to effectively support a decision regarding competing technologies • Complete a comparative evaluation of available commercial technology products and capabilities. • Recommend a technology NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

18. An Example—Analysis of the Problem • Question #2: “How will I know when I’ve solved it?” • Answer: When I can identify the quality of service, or the delivery conditions that must be met, to make each result acceptable • Complete a comparative evaluation… • The evaluation shall identify the top five technologies and vendors, based on market share. • The evaluation criteria for the analysis shall include all program objectives, program constraints, and technical requirements. • The evaluation shall provide a detailed mapping of each product/capability against the evaluation criteria. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

19. An Example—Analysis of the Problem • Question #2: “How will I know when I’ve solved it?” • Answer: When I can identify the quality of service, or the delivery conditions that must be met, to make the results acceptable • Recommend a technology… • The recommendation shall provide a rationale for the selected technology. • The recommendation shall provide a detailed analysis of the estimated lifecycle cost of the technology. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

20. The Resulting Performance Work Statement For Our Example NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

21. Additional Standards May Also Be Applied, As Necessary… • Workmanship • E.g., The final report shall contain no more than two (2) spelling or grammatical errors per every 5 pages. • Timeliness • E.g., The final report shall be delivered Not Later Than 180 days from the date of signature on the work authorization document. • Quantity • E.g., The contractor shall deliver five (5) copies, printed and bound, of each deliverable document. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

22. Subjectivity Is A Fact of Life In PBSA For Professional Services Subjective: adj. Peculiar to a particular individual, e.g., personal judgment. • The professional’s creative energies produce a result that is tailored to a specific context…they must suit a particular purpose. • That purpose is usually defined as the set of customer expectations. • For professional services, the performance standards—the conditions that make the results acceptable—are synonymous with the customer’s expectations. NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

23. Bounded vs Unbounded Subjectivity • Unbounded Subjectivity: personal judgment with no declaration or agreement on the properties to be judged or the evaluation criteria to be used • Ex. You arrive at a new car dealership and announce “I need a car…guess which one?” • Bounded Subjectivity: personal judgment based on agreed upon properties and criteria • Ex. You arrive at a new car dealership and announce “I need a family car and legroom, good gas mileage, and value are important to me. And I like the color blue. But not light blue.” • A subjective performance standard is still measurable! NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

24. How Well Did Our example Illustrate the Discussion? • In this sample problem, the specific services required were • Non-repetitive…this job will only be performed once • Context sensitive…similar services provided to a different client will produce different performance criteria • We have avoided any discussion of how the work will be performed, focusing instead on how well the work must be accomplished • Our performance criteria answered the Two Questions • We can define performance standards that define our bounded expectations NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

25. Tips For Implementing PBSA for Professional Services • Decide that it can be done • Translate objectives into a set of necessary results • Identify the criteria that make the results acceptable • Make the definition of performance criteria as simple and straightforward as possible • When subjectivity is necessary, strive for bounded subjectivity • Don’t be afraid of making mistakes…you will. But you will also learn from them • Remember, the simplest solution is usually the best one NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”

26. For More Information, Contact… • Mike Cameron • Cameron_mike@bah.com • (301) 543-4432 • Or send a general question or comment to pbsa@bah.com NCMA World Congress 2005 “Prime Time: Contract Management at the Core of the Enterprise”