Risk-Based Surveillance and the Quality Assurance Surveillance Plan (QASP)

1. RISK-BASED SURVEILLANCE Risk-Based Surveillance and the Quality Assurance Surveillance Plan (QASP)

2. Purpose • PURPOSE: To provide Quality Assurance Surveillance Plan (QASP) development and implementation guidance based on: • Procurement Information Circular No. 02-17, Government Quality Assurance Surveillance Plan (QASP) Guidance

3. Risk Management in the Acquisition Process Requirements Contract Award Acquisition Management Acquisition Strategy Solicitation Development Contract Incentives Source Selection Post Award Reward

4. Risk-Based Acquisition Categories Schedule Safety Risk Management Cost Environmental Security Technical Performance 4

5. Risk-Based Surveillance (RBS) Performance-Based Contracting FAR 37.601 • Performance-based contracting methods are intended to ensure that required performance quality levels are achieved and that total payment is related to the degree that services performed meet contract standards. Risk-Based Surveillance Surveillance enables us to monitor high risk areas, and establishes monetary payments based on how well risks are addressed.

6. BACKGROUND • Agency-wide assessment of NASA's Performance-Based Contracting for the enhancement of surveillance guidance. (http://ec.msfc.nasa.gov/hq/library/PBC.Team.Rpt.html) • Inspector General Inspection Report G-02-011 dated July 2, 2002 recommended that guidance be issued for technical/procurement personnel of regulatory requirements/policies regarding QASPs implementation/utilization for service contract. • New PIC gives information/guidance on QASPs. • Draft NASA Procedures and Guidelines for Surveillance Plans released for NODIS review.

7. Procurement Review • “A team from the Office of Procurement performed an assessment of PBC implementation by reviewing contracts and conducting interviews at nine NASA Centers. The team found that NASA has made significant progress in implementing PBC.” • “However, substantial areas for improvement in PBC implementation and management remain. Many of the areas for improvement are related to the reluctance observed throughout NASA to fully adopt PBC.” • “The team observed a lack of contract surveillance plans at many NASA centers. …. a surveillance plan based on risk management (programmatic risk) is an essential element of PBC.” NASA-wide PBC Assessment: Final Report, Updated  October 2000 http://ec.msfc.nasa.gov/hq/library/PBC.Team.Rpt.html [200209120/11:18EDT]

8. Inspector General Report • Inspector General Report that NASA is under-performing on Surveillance. • “We found that the development, content, and utilization of QASPs for NASA contracts could be significantly improved. Specifically, we found several weaknesses such as required plans not being developed, plans in place that are deficient in terms of content, and plans that are not being effectively utilized. In addition, for many of the contracts reviewed, we found a lack of documentation that would demonstrate that contract surveillance is being conducted systematically and consistently with the respective QASPs.” Review of Performance-Based Service Contract Quality Assurance Surveillance Plans, G-02-011, Final Report, June 24, 2002

9. Inspector General Report+ • Effective government contract surveillance, based on risk management, is essential aspect of PBC. • Surveillance plans not being developed for 36% of contracts reviewed. • Plans reviewed were vague and ambiguous. • Majority of plans did not address all required elements of the FAR. • Training is needed.

10. 1. What is a (QASP)? • A Quality Assurance Surveillance Plan defines what the Government will do to ensure that contractor performance is in accordance with contract requirements and related performance standards. • Government surveillance activities defined in the QASP can range from a one-time test or inspection of a product or service to periodic in-process monitoring of on-going contract performance.

11. 1. What is a QASP?+ A Quality Assurance Surveillance Plan provides for: • Observing, and documenting performance • Method for accepting contractor’s output • Formulas for calculating payments due • Adjustment in response to contractor performance, changed circumstances

12. Range of QASPs Supply Delivery Contract Quality Assurance SURVEILLANCE PLAN (QASP) BASELINE SFOC Contract Quality Assurance SURVEILLANCE PLAN (QASP) BASELINE

13. Why is a QASP Needed? • It is needed to ensure the Government receives the quality of supplies or services called for under the contract, pays only for the acceptable level received, and to assure that appropriated funds are spent wisely.

14. Why is a QASP Needed?+ • The Government has a duty to assure that appropriated funds are spent wisely.1 • That duty is fulfilled in part through surveillance.2 • Agencies shall develop quality assurance surveillance plans when acquiring services.3 Quality assurance surveillance plans (QASP) should be prepared in conjunction with the preparation of the statement of work.4 [1] NFS 1846.000 Scope of part [2] NFS 1846.000 Scope of part [3] FAR 37.602-2 Quality assurance. [4] FAR 46.401 General.

15. Why is a QASP Needed?+ • The quality assurance surveillance plan (QASP) which the project office prepares in conjunction with the statement of work is preliminary. It reflects the Government's surveillance approach relative to the perceived programmatic risk, and is written at a general rather than specific level because the risks will not be completely identified at that time. After contract award, contracting officers shall ensure that the QASP is revised to reflect the risks associated with the successful proposal. NASA FAR Sup Subpart 1846.4--Government Contract Quality Assurance 1846.401(a)

16. 2. What Types of Contracts Require QASPs? • Performance-Based Contracts (FAR 37.601(b)) • Service Contracts (FAR 46.103(a)) • Other Types of Work (FAR 46.401) • Government contract quality assurance surveillance is not limited to just performance-based and service contracts.1 [1] It may also apply to other types of contracted work. FAR 46.401(a) states that Government Contract quality assurance shall be performed at such times (including any stage of manufacture or performance of services) and places (including subcontractor’s plants) as may be necessary to determine that the supplies or services conform to contract requirements.

17. 2. What Types of Contracts Require QASPs?+ • Maintenance, overhaul, repair, servicing, rehabilitation, salvage, modernization, or modification of supplies, systems, or equipment. • Routine recurring maintenance of real property. • Housekeeping and base services. • Advisory and assistance services. • Operation of Government-owned facilities/systems • Communications services. • Architect-Engineering/Research & Development • Transportation and related services.

18. 3. How Does a QASP Help Us? • Provides a baseline for project monitoring and scheduling. • Provides a mechanism for assessing and assuring contract compliance, and the results help to identify areas requiring corrective actions. • Provide a mechanism for identifying, assessing, and documenting risks of contract performance, as well as tracking and mitigating risks.

19. 4a. Who Prepares a QASP? • The technical office responsible for the requirements prepares and maintains the QASP (FAR 46.103). • For most NASA requirements, the COTR is delegated the responsibility via the NASA Form 1634 to establish and deliver a QASP to the Contracting Officer.

20. 4a. Who prepares a QASP?+ • Surveillance should be a collaborative integrated effort that includes consideration of all areas of contract management (engineering, quality, procurement, finance, property, environmental, export control, safety and health, and other functional organizations) as appropriate.

21. 4b. When is a QASP Developed? • NFS 1846.401requires the development of a preliminary QASP in conjunction with the SOW. • The preliminary QASP reflects the Government's surveillance approach relative to the initially perceived programmatic risk, and is written at a general rather than specific level. • After contract award /prior to the start of contractor performance, the contracting officers shall ensure that the responsible technical office revises the preliminary QASP to reflect the risks that the SEB has identified as being associated with the successful proposal.

22. Contract STATEMENT OF WORK Draft Performance- Based Contract XYZ Services SOW/WBS Contract Quality Assurance SURVEILLANCE PLAN (QASP) Draft Contract Quality Assurance SURVEILLANCE PLAN (QASP) BASELINE QASP Flow Chart STATEMENT OF WORK Identifies work to be accomplished Describes work as “What”, not “How” PERFORMANCE-BASED CONTRACTS Requirements in terms of results Measurable performance standards Procedures for Reducing fees or price Performance Incentives where appropriate Ref: NFS 1846.401 General. FAR 37.602 Elements of PBC QUALITY ASSURANCE SURVEILLANCE PLAN After contract award, contracting officers shall ensure that the QASP is revised to reflect the risks associated with the successful proposal. 

23. 5. What Should a QASP Contain? • Surveillance schedule with clearly stated surveillance method(s) to be used. • Detail in the QASP regarding specific work requirements should be consistent with the importance and risk associated with that requirement.

24. 5. What Should a QASP Contain? • The QASP shall recognize the responsibility of the contractor to carry out its quality control obligations and shall contain measurable acceptance criteria corresponding to the performance standards contained in the statement of work. • The QASP shall focus on the level of performance required by the SOW, and associated performance requirements and standards (e.g. quality, quantity, and timeliness, etc.).

25. Risks from SOW/WBS Performance Analysis • For each SOW/WBS task and deliverable, review for potential risks. Then identify and develop risks, performance indicators and associated standards. • These must be aligned with the overall scope of the requirement and embody the criticality of the service and minimum needs. • Indicators and standards must be results-oriented.

26. Risk management is an ongoing process throughout the program. Risk analysis characterizes the probability and severity of identified risks. Risk management principles are applied as a decision-making tool. What are the technical risks associated with this WBS/SOW? What are the performance risks associated with this WBS/SOW? Risk Management and Analysis

27. Introduction Key Objectives Responsibilities Approach Objectives Methodology Documentation COTR/Inspector Schedule Actual Surveillance Surveillance Tools Informing Contract Mgr. Contract Discrepancy Report Reduced Inspections Surveillance by Customer Complaint Revisions to the QASP Appendices Acronyms/Definitions List of Deliverables Surveillance Tools Customer Feedback Mgt Information Sys. Checklists Surveys In-Depth Observation Performance Areas for Reactive Surveillance Example Detailed QASP Outline

28. 6. What Surveillance Methods? • Test/Inspection • Process/System/Product Monitoring • Management Information System • Checklist/Audit • Customer Input (Feedback) • Others

29. SOW Items Under Surveillance Contractor-provided metrics Contractor monthly reports Technical Interchange Meetings Milestone reviews Inspections Test reports Financial reports Failure reports Sampling Data Surveys Others? Determine best surveillance method

30. Test/Inspection • Appropriate for infrequent work (e.g. one time fabrication to print) /work with stringent performance requirements where safety or health is a risk. • Contractor performance verified by the Government at appropriate stages within development, mfg., delivery of product or service. • Intervals can range from mandatory test/ inspection, to periodic sampling depending on: • Risk related to the product/service delivered. • Demonstrated performance of the contractor.

31. Process/System/Product Monitoring • Monitoring of contractor critical process, system, and or product performance is a surveillance method. • This type of monitoring is typically conducted to supplement test and inspection results, however, it can also be used in lieu of test and inspection where appropriate. • This monitoring can be achieved through the evaluation of contractor provided performance metrics, a periodic independent review of contractor performance conducted by a designated Government representative.

32. Management Information System • Management Information Systems: Insight into contractor performance is by assessment of contractor generated data. In most cases, the contractor will already be generating the data in order to manage his processes. • Examples • Problem Reporting and Corrective Action data • Shop Floor Control/Data Collection System data • Work Authorization Document data • Maintenance Trend Analysis Report (MTAR)

33. Customer Input (Feedback) • Customer feedback is a reactive tool based on input from the customers with the primary purpose to provide performance feedback to the COTR. • This tool may be used as an indicator to increase government surveillance through use of different surveillance tools. • Customer feedback will generally not be the only tool used for critical processes and activities.

34. Example Details of Surveillance Tasks • Performance Work Statement • Surveillance Task • Customer Performance Evidence • Frequency • Pass/Fail or AQL

35. Indicators and Standards Key questions: what, how well, how many, when Performance Indicator Essential characteristic of acceptable performance and a feature of output that can be measured.

36. Performance Work Statement or SOW Par. IF A PERFORMANCE WORK STATEMENT IS USED IT SHOULD CONTAIN: • A statement of the required services in terms of outcome • A measurable performance standard for each outcome • A Pass Fail or Acceptable Quality Level for each outcome.

37. Surveillance Task • Some form of surveillance is necessary to determine the acceptability of the output. • The method and extent of surveillance will depend on what is minimally necessary to ensure the contractor complied with the performance standard or met the acceptance criteria.

38. Customer Performance Evidence • Can information on contractor work be acquired from a contractor system (preferably web based and on line)? • What report Needs to be Generated? • Management information system reporting? • Government Access? • Government Training to use?

39. Frequency/ Pass/Fail • How often should the Surveillance Activity be Performed? • What is the criteria for evaluation the contractor as having successfully performed the activity during the reporting period? • What constitutes failure? Discuss with the contractor in advance. • Have a resolution method.

40. Surveillance Examples

41. Surveillance Examples+

42. Acceptable Quality Level (AQL) • AQL is the performance measure which specifies acceptance procedures and is based on a sampling plan. • AQL chosen in accordance with the risks assumed • Formal methods using sampling plans e.g. Mil-Std-105 or ISO sampling plans

43. Sampling Surveillance Plan Example Take Sample Accept/Reject* Based on Acceptable Quality Level ALL SOW TASKS • . • . • . • . • . • . • . • . LOW HIGH 500 items 8 samples Mil-STD-105E Sampling Master Table Plan*

44. 7. Surveillance Coordination Between the Government and Contractor? • Contractors should be briefed on surveillance requirements and responsibilities at a post-award conference. • Surveillance methods to be used should be discussed to confirm they are fully understood. • Reviewing and discussing the contractor's plan for maintaining an acceptable quality level under the contract is important. • NOTE: Although communication regarding surveillance occurs between the Government /the contractor, the Gov. still has QASP responsibility.

45. 8. Do Surveillance Activities Ever Change During the Contract? • The QASP should be periodically reviewed once performance has begun to determine if changes in surveillance levels or methods are necessary, and updated accordingly. • When a change in the required work (e.g. change of design) or change in work performance (a supplier assessment or negative trend) results in a change in risk, potential surveillance changes should be considered.

46. Do Surveillance Activities Ever Change During the Contract?+ • Where surveillance results show good performance consistently, the amount of surveillance should be adjusted accordingly. This saves the government money, reduces oversight burdens on the contractor, and recognizes the contractor's achievement of performance. • After contract award, contracting officers shall ensure that the QASP is revised to reflect the risks associated with the successful proposal.1 • When performance is deficient, the contracting officer should notify the contractor promptly, track corrective actions, and consider appropriate surveillance adjustments. [1] FAR Subpart 1846.4--Government Contract Quality Assurance 1846.401General.

47. 9. Safety • While developing a strong surveillance approach to contract structure, never lose sight of the fact that safety is NASA's highest value.

48. CONCLUSIONS

49. Surveillance Depends On Increasing Risk Low Risk High Risk • Surveillance depends on defining high and low risk: • Technical risk levels • Evaluation of contractor’s capability/ past performance • How well processes are defined • Program visibility and impact of failure • Design/manufacturing complexity, producibility • Value of asset • Prior Inspections

50. Risk-Based SURVEILLANCE RBS PROCESS • RFP Participants / Quality / Engineering • Evaluate Risks from all Sources. • Revise Ranking Risks as a new set. • Propose Mitigation Strategies • Evaluate Mitigation Strategies involving Surveillance • Revise Surveillance Plan Framework • Incorporate Mitigation Strategies into Plan