IMPROVING AUDITS Nile A. Luedtke DOECAP Operations Team ASP Workshop 2011 Pleasanton, California

1. IMPROVING AUDITS Nile A. Luedtke DOECAP Operations Team ASP Workshop 2011 Pleasanton, California

2. Audit Goals To Perform Meaningful Audits To Clearly Communicate Audit Results To Confirm Completion of Corrective Actions and Finding Closures To Promote Effective Corrective Actions Continuous Improvement

3. Basic Audit Stages • Audit Preparation • Audit Execution • Audit Reporting

4. Audit Preparation • Review DOECAP Requirements • Facility Document Reviews • Quality Assurance Plans • Standard Operating Procedures • Radiation Control Plans • Health & Safety Plans • Etc. • Review Facility CAP and CAP Update

5. Audit Preparation • Review Your DOE Site Specific • Contract Requirements • Facility Interactions and Performance • Review TSDF Permits • Perform the Agency Review for TSDFs • Participate in Pre-Audit Team Call • Ensure Your Auditor Training is Complete and Up-To-Date

6. Audit Preparation • Proficiency Test Performance Review • Two consecutive failures • Consistent bias • Cross-matrix analyte failures • Sensitivity errors (false negatives, false positives) • Analyte identity errors • Review and reporting errors (incorrect units, decimal shifts, data entry)

7. Audit Preparation • Start Using the DOECAP Checklists • Complete LOIs and areas of the Checklists based on facility document reviews • Add LOIs to the Checklists based on facility document reviews, your contracts, and information from the previous audit • Add your Site information regarding facility performance

8. Audit Execution • Takes More Time When Not Prepared • Need To Use Time On-Site To Audit • Continuing Audits Need to Build on Previous Audits • Audit Specific Technical Processes and Their Implementation • Deficiencies at this level may lead to broader systematic and programmatic issues

9. Audit Execution • Focus On: • Personnel Interviews (use open questions, let them do the talking) • Direct Observation (have them demonstrate or watch them perform actual operations) • Data and Documentation Review (always ask them to provide documentation, all auditors are from Missouri, “SHOW ME”) • Indirect Observation (keep your eyes and ears open, pay attention to items not associated with your current LOI)

10. Audit Execution • Look For: • SOP vs. Actual Facility Compliance • Facility Practices vs. Regulatory and Analytical Method Requirements • Old Finding Closures or Non-Closures • New Finding Documentation • Observation Documentation and Rationale • Priority I Finding Requires Immediate Interaction with Lead, Facility, and DOECAP Management

11. Audit Documentation • Completed Checklists • Findings & Observations • Old Finding Closures/non-Closures • Section Narratives • Executive Summary Paragraphs • List of Personnel Interviewed • List of Referenced Facility SOPs

12. Writing a Finding • One Sentence Finding Statement • Identify Requirement Reference (QSAS, CFR, Permit, Facility Document or SOP) • Finding Narrative • State or paraphrase the requirement • Identify what is done, what is not done, or what is done incorrectly • Discuss additional details, contributing factors, examples, or direct evidence

13. Example Finding QA Finding-A: Over the past several months, the laboratory data review and reporting processes have exhibited significant deficiencies relative to PT submissions. (Priority II) (QSAS Rev. 2.6, Section 4.2.3.s and 4.2 DOE-3; Laboratory QAP Sec.11) The QSAS requires laboratories to establish data review processes and ensure that data review is inclusive of all quality related steps. The laboratory QAP identifies a three-tiered data review process and individual SOPs contain review steps and data review checklists. However, reporting discrepancies and failures identified in multiple PT analyses indicate the process is not fully understood or implemented. Examples include: PT HW0710-SOIL (September 2010) – All explosive compound results were not acceptable and values were significantly elevated relative to the target value and acceptance window. Reported values indicate a major calculation error or a possible decimal shift.

14. Writing an Observation • Deviations from Best Management Practices, non-requirement-based issues, or an opportunity for improvement, which warrant attention • Must clearly state the issue (so what? test) • Identify the bad things that could happen • Deviations from “requirements” need to be identified as findings, even if they might be “isolated instances”

15. Writing Narratives • Section Narratives • This is a short concise overview • One to two pages are sufficient • If you need to convey more information to next year’s auditor, use the Checklist • Executive Summaries • One paragraph on your area of review • Identify and paraphrase each finding

16. Audit Report Review Draft Review During Audit (prior to audit close-out help by reviewing other auditors findings, observations, and narratives) Draft Review During Facility Factual Accuracy Period (the week following the audit take one more look) Operations Team and DOE Management Review Period (respond to questions and help with clarifications)

17. Corrective ActionsReview Will the Actions Fix the Problem? Will the Actions Correct the Deficiency Across Related Systems, Processes, and Methods? Will the Actions Prevent the Problem from Reoccurring? Facility – Auditors – Ops Team Need to Reach A Consensus on Actions

18. Corrective ActionsAcceptance When the Corrective Actions are considered acceptable, each auditor must confirm acceptance with the DOECAP Corrective Action Coordinator The Process is not complete until the proposed actions have been reviewed and determine to be acceptable

19. QUESTIONS