Creating a Framework for Data Quality in AF Performance Based Remediation Contracts

1. Creating a Framework for Data Quality in AF Performance Based Remediation Contracts

2. Agenda Background DoD Initiatives AFCEE programmatic and project quality PBR contracting language AQA presentation 2

3. What is data quality?Why do we care? 3 Project requirements – what data do we need and how good does it need to be? How good is the lab? How “good” is the data? Project decisions – can the data support our decision? (e.g., site closeout) Project Requirement – generally determined through the Data Quality Objective Process (EPA QA/G4, Feb 2006); identified in the project Work Plan (UFP-QAPP) State the problem Identify the goal of the study Identify information inputs Define the boundaries of the study Develop the analytic approach Specify performance or acceptance criteria Develop the plan for obtaining data Data quality indicators (PARCCS) Precision – refers to the reproducibility of measurements. Accuracy – degree of agreement between an observed value and a “true” value. Surrogate recoveries assess accuracy as do PT samples. Representativeness – refers to the degree in which data accurately and precisely depict the characteristics of the population. Completeness – a measure of the amount of valid data compared with the amount that was expected to be obtained under correct, normal conditions. Comparability – a qualitative indicator of the confidence with which one data set can be compared to another data set. Sensitivity – the ability of the analytical method or instrument to discriminate between measurement responses representing different concentrations. Project Requirement – generally determined through the Data Quality Objective Process (EPA QA/G4, Feb 2006); identified in the project Work Plan (UFP-QAPP) State the problem Identify the goal of the study Identify information inputs Define the boundaries of the study Develop the analytic approach Specify performance or acceptance criteria Develop the plan for obtaining data Data quality indicators (PARCCS) Precision – refers to the reproducibility of measurements. Accuracy – degree of agreement between an observed value and a “true” value. Surrogate recoveries assess accuracy as do PT samples. Representativeness – refers to the degree in which data accurately and precisely depict the characteristics of the population. Completeness – a measure of the amount of valid data compared with the amount that was expected to be obtained under correct, normal conditions. Comparability – a qualitative indicator of the confidence with which one data set can be compared to another data set. Sensitivity – the ability of the analytical method or instrument to discriminate between measurement responses representing different concentrations.

4. 4 Background Inspector General Audits (1996 – 1997) DoD IG Findings “Organizations reviewed were not efficient and effective in the procurement and administration of environmental testing contracts” EPA IG Findings “EPA had developed many critical elements necessary for a strong and effective QA program BUT “Managers had not demonstrated commitment to fully develop and implement the program” DoD IG Audit Objectives: Evaluate cost and operational effectiveness of lab support services provided for environmental testing. Determine if DoD organizations were effectively managing contracts and effectively performing QA procedures. DoD IG Recommendations: Develop standardized quality assurance procedures Improve laboratory contracting practices Include remedy clauses in environmental testing contracts US District Court, Sep 2000 Indictments against 13 employees of Intertek Testing Services Conspiracy to commit mail fraud; conspiracy to present false, fictitious, and fraudulent claims against the United States; mail fraud, present false, fictitious, and fraudulent claims against the United States; wire fraud Five plead guilty, four to misdemeanors; none served jail time DoD IG Audit Objectives: Evaluate cost and operational effectiveness of lab support services provided for environmental testing. Determine if DoD organizations were effectively managing contracts and effectively performing QA procedures. DoD IG Recommendations: Develop standardized quality assurance procedures Improve laboratory contracting practices Include remedy clauses in environmental testing contracts US District Court, Sep 2000 Indictments against 13 employees of Intertek Testing Services Conspiracy to commit mail fraud; conspiracy to present false, fictitious, and fraudulent claims against the United States; mail fraud, present false, fictitious, and fraudulent claims against the United States; wire fraud Five plead guilty, four to misdemeanors; none served jail time

5. DoD Reaction AFCEE Quality Assurance Project Plan, 1997 DoD Quality Systems Manual for Environmental Laboratories, Mar 2000 Intergovernmental Data Quality Task Force (EPA, DoD, DOE), 1997 Uniform Federal Policy for Implementing Environmental Quality Systems, (UFP-QS) Mar 2005 Uniform Federal Policy for Quality Assurance Project Plans, (UFP-QAPP) Mar 2005 DoD Instruction 4715.15 – Environmental Quality Systems, Dec 2006 DoD Policy and Guidelines for Acquisitions Involving Environmental Sampling and Testing, Nov 2007 DoD Environmental Laboratory Accreditation Program, Dec 2008 5 AFCEE QAPP - This detailed QAPP has been prepared as a guide for use by contractors providing environmental services to AFCEE. It is designed to facilitate the preparation of site-specific QAPPs (hereafter referred to as project QAPPs) and to establish the analytical protocols and documentation necessary to ensure that data are generated, reviewed, and assessed in a consistent manner. DoD QSM - The Department of Defense (DoD) Environmental Data Quality Workgroup (EDQW) developed this manual to provide baseline requirements for the establishment and management of quality systems for environmental testing laboratories performing services for the Department of Defense. It is based on the National Environmental Laboratory Accreditation Conference (NELAC) Chapter 5 Quality Systems standard (July 1999), and it also incorporates the requirements of ISO/IEC 17025:2005. DoD-specific requirements, clarification of requirements, and guidance for implementation are contained in numbered gray boxes. IDQTF - Comprised of EPA (headquarters and Regions), Department of Defense, and Department of Energy representatives, the focus of the IDQTF is to comprehensively address a myriad of problems and issues related to the management of environmental data quality at Federal facilities. UFP-QS – Outlines essential elements of a QS for management of environmental data collection and use and environmental technology programs UFP-QAPP – Acts as a single national consensus guidance document to implement ANSI/ASQ E4, Quality Systems for Environmental Data and Technology Programs – Requirements with guidance for use, Section 6 (Part B) (which requires that a QAPP be approved for all data collection projects) DoDI 4715.15 - Responsibilities & procedures for implementing environmental QS into all activities and programs involving collection/use of env data; Provides for Environmental Data Quality Workgroup; Authorizes formal publication of certain intergovernmental quality systems documents as DoD Manuals DoD Policy for Sampling and Testing - Describes roles & responsibilities of Govt/Contractor chemist; Invokes the QSM, UFP-QS, UFP-QAPP; Provides language to incorporate into a solicitation DoD ELAP – Established that labs conducting testing for restoration program must be accredited; Accreditation based on conformance with DoD QSM. For AFCEE contracts, a lab pre-audit is no longer necessary, but primes still have to determine if lab can satisfy project requirements. AFCEE QAPP - This detailed QAPP has been prepared as a guide for use by contractors providing environmental services to AFCEE. It is designed to facilitate the preparation of site-specific QAPPs (hereafter referred to as project QAPPs) and to establish the analytical protocols and documentation necessary to ensure that data are generated, reviewed, and assessed in a consistent manner. DoD QSM - The Department of Defense (DoD) Environmental Data Quality Workgroup (EDQW) developed this manual to provide baseline requirements for the establishment and management of quality systems for environmental testing laboratories performing services for the Department of Defense. It is based on the National Environmental Laboratory Accreditation Conference (NELAC) Chapter 5 Quality Systems standard (July 1999), and it also incorporates the requirements of ISO/IEC 17025:2005. DoD-specific requirements, clarification of requirements, and guidance for implementation are contained in numbered gray boxes. IDQTF - Comprised of EPA (headquarters and Regions), Department of Defense, and Department of Energy representatives, the focus of the IDQTF is to comprehensively address a myriad of problems and issues related to the management of environmental data quality at Federal facilities. UFP-QS – Outlines essential elements of a QS for management of environmental data collection and use and environmental technology programs UFP-QAPP – Acts as a single national consensus guidance document to implement ANSI/ASQ E4, Quality Systems for Environmental Data and Technology Programs – Requirements with guidance for use, Section 6 (Part B) (which requires that a QAPP be approved for all data collection projects) DoDI 4715.15 - Responsibilities & procedures for implementing environmental QS into all activities and programs involving collection/use of env data; Provides for Environmental Data Quality Workgroup; Authorizes formal publication of certain intergovernmental quality systems documents as DoD Manuals DoD Policy for Sampling and Testing - Describes roles & responsibilities of Govt/Contractor chemist; Invokes the QSM, UFP-QS, UFP-QAPP; Provides language to incorporate into a solicitation DoD ELAP – Established that labs conducting testing for restoration program must be accredited; Accreditation based on conformance with DoD QSM. For AFCEE contracts, a lab pre-audit is no longer necessary, but primes still have to determine if lab can satisfy project requirements.

6. 6 EPA has identified/recommended five Quality Assurance methods as being the most effective: Data Validation Double blind PE Lab audit Magnetic tape audit Split samples Additionally, EPA identified 3 of the least effective methods: Single blind PE Lab certification Past performance In the DoD IG’s 1997 Audit, AFCEE was found to be employing Data Validation, Double Blind PE, and Lab Audit (Most Effective) as well as Lab certification and Past performance (Least Effective) Laboratory Performance Evaluation Samples Single-blind or double-blind proficiency testing Assesses internal processes, analytical capabilities of the lab Poor performance may indicate need for more in-depth assessment PE samples a key component of 4-Base PBC Direct Laboratory Assessment When? Questionable results from ELAP inspection or PE occur Usually in conjunction with prime contractor technical staff Team reports findings, recommends corrective action EPA has identified/recommended five Quality Assurance methods as being the most effective: Data Validation Double blind PE Lab audit Magnetic tape audit Split samples Additionally, EPA identified 3 of the least effective methods: Single blind PE Lab certification Past performance In the DoD IG’s 1997 Audit, AFCEE was found to be employing Data Validation, Double Blind PE, and Lab Audit (Most Effective) as well as Lab certification and Past performance (Least Effective) Laboratory Performance Evaluation Samples Single-blind or double-blind proficiency testing Assesses internal processes, analytical capabilities of the lab Poor performance may indicate need for more in-depth assessment PE samples a key component of 4-Base PBC Direct Laboratory Assessment When? Questionable results from ELAP inspection or PE occur Usually in conjunction with prime contractor technical staff Team reports findings, recommends corrective action

7. Transition to UFP-QAPP: Deliverables 7 UFP-QAPP section 1.2: “The QAPP document may be referred to by another name, including field sampling plan (FSP), workplan, sampling and analysis plan (SAP). The UFP-QAPP integrates ALL technical and quality aspects for the life cycle of the project.” For contractors, they should work with Contracting Officer’s Representative to make sure deliverable requirements are met. This diagram shows both content and document deliverables (both to AFCEE and to regulators). Can also look at this diagram as a process. Model plans were formats as well as content; UFP-QAPP is a format that requires specific content. Model QAPP is being replaced with QA/QC compendium and QSM requirements. UFP-QAPP section 1.2: “The QAPP document may be referred to by another name, including field sampling plan (FSP), workplan, sampling and analysis plan (SAP). The UFP-QAPP integrates ALL technical and quality aspects for the life cycle of the project.” For contractors, they should work with Contracting Officer’s Representative to make sure deliverable requirements are met. This diagram shows both content and document deliverables (both to AFCEE and to regulators). Can also look at this diagram as a process. Model plans were formats as well as content; UFP-QAPP is a format that requires specific content. Model QAPP is being replaced with QA/QC compendium and QSM requirements.

8. PBR Contract Language 5.0 CHEMISTRY REQUIREMENTS  The Contractor shall be responsible for the quality of all required chemistry services performed. The Contractor shall ensure that all chemistry-related tasks are conducted in accordance with the project-specific Work Plan (WP). The WP shall be written in the Uniform Federal Policy for Quality Assurance Project Plans (UFP-QAPP) format (i.e., worksheets) and shall include the sampling and analysis plan. The sampling and analysis plan, as described in the National Contingency Plan, 40 CFR 300.415, contains the required elements of a QAPP and Field Sampling Plan (FSP). Sampling and Analysis Plan (SAP), which includes the Quality Assurance Project Plan (QAPP) and Field Sampling Plan (FSP). The Contractor shall identify a Project Chemist as key personnel in the project SAPWP.  The Project Chemist will act as a POC on all chemistry-related issues and shall be responsible for ensuring that all Data Quality Objectives (DQOs) are met. 8 PBR contract language now refers to a Work Plan in the UFP-QAPP format, which includes the elements of the QAPP and Field Sampling Plan, as well as the Sampling and Analysis Plan. Previously, contracts called for a SAP, which included the QAPP and FSP. Contractors should now be producing a single deliverable, a Work Plan in the UFP-QAPP format. There is no need to produce separate SAP, QAPP, and FSP. PBR contract language now refers to a Work Plan in the UFP-QAPP format, which includes the elements of the QAPP and Field Sampling Plan, as well as the Sampling and Analysis Plan. Previously, contracts called for a SAP, which included the QAPP and FSP. Contractors should now be producing a single deliverable, a Work Plan in the UFP-QAPP format. There is no need to produce separate SAP, QAPP, and FSP.

9. PBR Contract Language 5.1 Quality Assurance The Contractor shall develop project-specific DQOs designed to ensure data of adequate quality are collected to support project decisions. DQOs shall be developed in accordance with local, state, and federal regulations, such as USEPA QA/G4, Guidance for the DQO Process (most recent version) and documented in the project SAPWP.  All field sampling and laboratory services shall be conducted in accordance with the approved project QAPPWP. All field sampling shall be conducted in accordance with the approved project FSP. Samples shall not be submitted for analysis until the SAP WP is approved. The Uniform Federal Policy (UFP)-QAPP serves as a guidance document in the development of the project specific QAPP. If an installation-wide basic UFP-QAPP has not yet been established for the installation, the Contractor is responsible for establishing the basic UFP-QAPP unless otherwise approved by the COR.  The Contractor shall ensure that all requirements specified in the project SAP WP are met. If not met, the Contractor may be required to re-accomplish sampling at the Contractor’s expense. The Contractor shall conduct audits, administer a performance evaluation sample program, verify and validate data, and perform corrective actions in accordance with the project SAPWP. The Contractor shall submit a Quarterly Lab Use Reportaudit results and performance evaluation samples to the COR . 9 Again, the Work Plan is the deliverable of interest in this modified language. Also, rather than submitting a Quarterly Lab Use Report, contractors should submit any audit results and performance evaluation sample results to the COR.Again, the Work Plan is the deliverable of interest in this modified language. Also, rather than submitting a Quarterly Lab Use Report, contractors should submit any audit results and performance evaluation sample results to the COR.

10. PBR Contract Language 5.2 Laboratory Selection The Contractor shall select a laboratory that is accredited under the DoD Environmental Laboratory Accreditation Program (ELAP). The Contractor shall ensure that the selected laboratory meets all state and federal requirements, including state certification where appropriate. The laboratory must also have analytical capabilities sufficient for the methods specified in the WP and adequate throughput capacity to handle the project’s analytical workload during all field activities. Please submit any questions regarding to ELAP to afcee.chemistry.support@us.af.mil. The Contractor shall select a laboratory that complies with the requirements of the current version of the DoD Quality Systems Manual (QSM). The current version of the QSM can be found at the following website: http://www.navylabs.navy.mil/. The laboratory must also have analytical capabilities sufficient for the methods specified in the SAP and adequate throughput capacity to handle the project’s analytical workload during all field activities. The Contractor shall ensure that the selected laboratory meets all state and federal requirements, including state certification where appropriate.   10 This language simply clarifies the DoD ELAP requirement and states that the laboratory must have the capabilities and capacity to meet the project requirements in the Work Plan.This language simply clarifies the DoD ELAP requirement and states that the laboratory must have the capabilities and capacity to meet the project requirements in the Work Plan.

11. PBR Contract Language 5.2 Laboratory Selection (cont) The Contractor may establish an on-site laboratory at the project site if determined necessary by the Contractor. However, on-site test laboratory(ies) shall also be accredited under the DoD ELAP and meet all state and federal requirements, including state certification where appropriate. The laboratory(ies) to be used by the Contractor shall be DoD Environmental Laboratory Accreditation Program (ELAP) certified or equivalent. The Contractor may establish an on-site laboratory at the project site if determined necessary by the Contractor. However, on-site test laboratory(ies) shall be DoD ELAP certified or equivalent and meet the requirements of USEPA, specific state regulator requirements, and all requirements of the most recently approved DoD QSM. 11 This language specifically addresses on-site labs (mobile labs). On-site labs must also be accredited under the DoD ELAP. Each of the accrediting bodies may handle the accreditation differently. For example, one AB may grant accreditation to an on-site lab if it follows the quality system and standard operating procedures of a parent, fixed laboratory, whereas another AB may grant accreditation to an on-site lab independently of the parent lab.This language specifically addresses on-site labs (mobile labs). On-site labs must also be accredited under the DoD ELAP. Each of the accrediting bodies may handle the accreditation differently. For example, one AB may grant accreditation to an on-site lab if it follows the quality system and standard operating procedures of a parent, fixed laboratory, whereas another AB may grant accreditation to an on-site lab independently of the parent lab.

12. Summary Standardized quality assurance & control practices: Help identify (and hopefully reduce) poor laboratory practice and fraud Increase our confidence in the analytical results Support good decision-making AFCEE has incorporated best chemistry practices both programmatically and on a project-specific basis AFCEE has incorporated language consistent with DoD initiatives into the WERC09 contract to support performance based remediation 12