DoD-ELAP Update

1. DoD-ELAP Update • Presented to: • DOE ASP Workshop 2012 • Idaho Falls, Idaho • Fred McLean • Navy Laboratory Quality & Accreditation Office • Chair, DoD Quality Assurance Oversight Subgroup

2. Agenda • Who we are: Environmental Data Quality Workgroup (EDQW) • The DoD Environmental Laboratory Accreditation Program (ELAP) • Accreditation: What are we finding now? • DoD/DOE Joint Initiatives: QSM & LCS Study

3. Environmental Data Quality WorkgroupCharter Issued 01 October 2010 Develop and recommend policy related to sampling, testing, and quality assurance for environmental programs to eliminate redundancy, streamline programs, improve data quality, and promote data integrity. Coordinate the exchange of information among DoD components. Develop DoD issuances to implement environmental quality systems and promote cost effective government oversight. Implement and provide oversight of the DoD ELAP.

4. Accreditation: What is necessary for “Stability”? Unified standard Trained assessors Uniform interpretation of the requisite standard Standardized assessment procedures What is necessary for “Better Communication”? A defined process to resolve technical differences Training for our own Project Managers Continued improvement of the procurement process 4

5. DoD ELAP – Status • September 2009: • DoD ELAP becomes an official program effective 01 October 2009 • Four third party Accreditation Bodies evaluated & chosen: A2LA; ACLASS; L-A-B; PJLA • September 2011: • 98 laboratories applied for DoD ELAP accreditation • 96 laboratories assessed • 84 laboratories accredited • September 2012: • 101 laboratories have been accredited • 38 States represented along with Canada & Italy 5

6. DoD ELAP ProgramRequirements DoD-wide program acceptance Applies to collection of definitive data for environmental restoration programs Laboratory compliance with DoD QSM Accreditation performed by our four third party AB’s EDQW perform oversight of program Projects select accredited laboratories 6

7. DoD ELAPAccreditation Issues (from a DoD perspective) • Labs not in compliance with QSM standard at the time of the assessment • Assessment consistency • Scope creep • Project specific approvals & deviations from the QSM 7

8. Accreditation Bodies (ABs)What are they finding now? • LOD/LOQ requirements • Quarterly verifications for all matrices, analytes • Calibration & standards certificates • ISO 17034 requirements for reference providers • Corrective Actions • Corrections vs. corrective actions, root cause analysis 8

9. Accreditation Bodies (ABs)What are they finding now? • Proficiency Testing (PT) issues • Lack of PTs for all methods/analytes requested by scope/projects • Measurement Uncertainty • Procedure(s) to properly calculate/report uncertainty • Method Modifications and Method Validation • Calibration model, interferences, selectivity, sensitivity, precision & bias 9

10. Accreditation Bodies (ABs)What are they finding now? • Fundamental ISO 17025 program requirements missing: • Lack of internal audits • No managerial type reviews • Limited data reviews • Documents (procedures and QA Manuals) out of date • Lack of traceability • Limited to no oversight by QA Officials 10

11. Better Communication Tools • We are striving for better communication among our labs, ABs, and ourselves. We have developed a new tool to help us that is based on a simple, two page survey rating from 1 (poor) to 5 (excellent) to gauge how the accreditation process is working: 1. Ease of finding information on the DOD ELAP 2. Ease of accessing EDQW through the DENIX website 3. Ease of finding and comparing the ABs • Policies and procedures are effectively communicated (Both QSM and AB) 5. The assessment team was well prepared prior to arrival 6. The length of the assessment was appropriate 7. The Assessor(s) knowledge of assessment standard(s) was thorough 8. The assessor(s) had appropriate technical and practical knowledge 9. The AB was available for clarification of findings 10. The time frame for the overall process was reasonable 11

12. Better Communication Tools • We are also interested in the frequency of change (from “never” to “more than once a year”) within a laboratory for critical issues such as: Additions to Scope of Accreditation Deletions from Scope of Accreditation Variations from the QSM are taken Limit of detection (LOD)s adjusted (after verified) Limit of quantitation (LOQ)s adjusted (after verified) LCS Acceptance Limits Updated Control Charts Monitored 12

13. DoD / DOE Joint Initiatives DoD QSM / DOE QSAS consolidation update: A “stand-alone” document has been created that requires ISO 17025 & 2009 TNI Standard for full compliance. Over 400 comments have been addressed and final edits will be sent to our contractor to create a final draft version. This final draft expected by 01 November 2012, at which time it will be ready for the signature cycle. Final document ready by the first of January, 2013. 13

14. DoD / DOE Joint Initiatives: LCS Study Original LCS Study: Based on data collected in December 2000 20 DoD approved labs asked to participate Most methods had 17 labs send data; with a few methods (such as Herbicides) only 5 sent data Labs were asked to send their last 20-30 LCS values for each method/analyte without censoring Control Limits were established at 3 standard deviations around the mean, except for Method 8151 (Herbicides) at the 5th and 95 percentiles 14

15. LCS Study New LCS Study: Based on data collected in summer 2012 All DoD-DOE accredited/approved labs asked to participate (approximately 120) Labs were asked to send their last years worth of data, up to three years if less than minimum 30 data points All information was captured through electronic data files Other important information such as prep methods and spiking levels were captured 15

16. LCS Study New LCS Study: Number of laboratories participating: 53 Number of records uploaded: 6.7 million Number of analytes: 1,003 Number of methods: 26, including two CLP 16

17. Summary LCS Data 17

18. Summary LCS Data 18

19. Data Challenges Fields were text strings: this allows variations in describing same analyte and method Determining methods from SOP names CAS IDs were not consistent in all cases Matrices mapped to Water, Solids, and Gases (matrices such as toy, filter, paper, or tissue not used) Mismatches between reported and calculated percent recovery (possible rounding/truncation issues) Incorrect units or proportions reported (99% recovery entered as 0.99; spikes at 50 µg/L with a reported recovery of 48 mg/L) Overall, less than 0.5% of all available data was removed from the study due to the above reasons 19

20. Remaining Questions What is our desired failure rate? What is considered a “good” control limit? What effect does prep method have on the results? Is this an analyte by analyte determination? Should LCS limits at least equal CCV limits? Is +/- 3 Standard Deviations the best approach? How are the new limits used? Accreditation/Approval? In-house vs. published? Trend analysis? 20

21. DoD ELAP & QSM Resources QUESTIONS??? • Published information found on websites: • https://www.denix.osd.mil/portal/page/portal/EDQW • www.navylabs.navy.mil 21

22. Improving Environmental Data Quality… Because the Right Decisions Require Quality Data • Fred.mclean@navy.mil • www.navylabs.navy.mil • www.denix.osd.mil/edqw/