2013 ASP Workshop & DOECAP Meeting

1. 2013 ASP Workshop & DOECAP Meeting Auditor/Lead Auditor Training for Laboratory and TSDF Audits What is the Requirement? Richard Martin, DOECAP Deputy Manager, Oak Ridge Office September 23, 2013 1

2. The Training Goals • DOECAP management’s goals: • Consistency • Requirement-based • Customer-oriented • The DOECAP audit reports have been getting better each year, and this training is designed to: • Make it EASIER to write solid, defensible findings. • Help you meet DOECAP’s goal of continuous improvement to the audit reports. 2

3. Change to Audit Report Review Process • Beginning August 28, 2013, DOECAP management instituted a 3-day audit team review of “almost final” audit reports. • When the DOECAP Manager is ready to approve the report, the Operations Team provides the report to the audit team for a final review. • Any final corrections identified by the audit team will be addressed before the report is approved. If there are any outstanding issues, DOECAP management coordinates and resolves them directly with the Auditor/Lead Auditor. 3

4. Professional Code of Conduct • Each Lead Auditor and Auditor is considered an official representative of DOE. • Always be professional: • Avoid adversarial relationships. • Avoid being critical of the organization or the facility personnel. • Temper negative comments during the course of the audit. • Be sensitive to the pressures and stress experienced by the people being audited. 4

5. Professional Code of Conduct (cont.) • During an audit, each team member must: • Follow the rules and requirements established by the audited facility. • Wear appropriate attire and PPE (shoes, safety glasses, etc.), as directed by the audited facility. • Avoid displaying a superior attitude or portraying yourself as an authority figure. • Refrain from excessive chatter, and avoid vulgar language or the use of flippant remarks. • Be discreet while socializing. 5

6. Conflict of Interest • DOE contractors and federal staff must avoid any conflict of interest or the appearance of a conflict of interest. • Should a potential conflict of interest be encountered, immediately report the issue to the Lead Auditor. • DOECAP’s credibility will be damaged if auditors violate the Code of Conduct or fail to speak up about a conflict of interest. 6

7. Step 1 – Identify the Deficiency What is the problem? Examples A: Didn’t find documentation of the weekly inspection of a storage area for the last week of December. B: Didn’t find a required calibration record. 7

8. Step 1 – Identify the Deficiency (cont.) Examples C: Observed a sampling step that was not addressed in a procedure. D: Training records for two employees showed overdue annual continuing demonstrations of capability. E: Observed personnel exposure monitoring using wrong device. 8

9. Step 2 – Identify the Requirement • Very Important • WHAT IS THE REQUIREMENT? • Focus on the specific requirement that is not being met. • The requirement could be based on multiple sources or a hierarchy of sources, such as a procedure, a method, a permit, and/or a regulation. • Do the research, within the audit time constraints. 9

10. Step 2 – Identify the Requirement Examples A: Permit requires that inspections of storage areas be performed weekly and documented on Form X. B: The QSAS requires instruments to be calibrated and the calibration documented. Procedure CAL-100 requires documentation in the logbook specific to the instrument. C: The QSAS and NQA-1 require all activities to be described in a procedure. 10

11. Step 2 – Identify the Requirement Examples D: Procedure TWY-101 requires employees to perform annual continuing demonstrations of capability. E: RML requires exposure monitoring to be performed in accordance with procedures submitted with the license application. Procedure RAD-100 requires use of Device A for exposure monitoring. 11

12. Step 3 – Pull the String • Within the time constraints of the audit, determine the extent of condition (pull the string) to help verify whether you have identified the correct requirement (regulatory versus QA) and whether it appears to be programmatic. • Notify the Lead Auditor that you have an issue. • If it’s a QA issue, notify the QA auditor as well. 12

13. Step 4 – Write the Requirement Statement • Be specific to the issue. Pick the requirement that zeros in on the deficiency, especially if you reference multiple sources. • Pare it down to focus on only the part of the requirement that is relevant to the deficiency. • Avoid including words or phrases from irrelevant parts of the requirement(s). 13

14. Step 4 – Write the Requirement Statement (cont.) • Don’t try to justify or prove the applicability of the requirement. • Don’t restate the source of the requirement. • Keep it short, and get to the point! 14

15. Step 4 – Write the Requirement Statement (cont.) Examples A:Storage area inspections must be documented on Form X. B:Calibrations for Instrument AB must be recorded in Logbook IAB. C:Tank sampling must be described in a procedure. 15

16. Step 4 – Write the Requirement Statement (cont.) Examples D: Employees must perform annual continuing demonstrations of capability. E: Device A must be used to perform exposure monitoring. 16

17. Step 5 – Write the Deficiency Statement & Discussion • The Deficiency Statement should match the Requirement Statement, using the same words and phrases whenever possible. • The discussion provided after the Deficiency Statement “tells the story.” Explain how you obtained the information (i.e., how do you know), the circumstances, and any mitigating factors. 17

18. Step 5 – Write the Deficiency Statement & Discussion (cont.) Example A:Form X was not used to document the storage area inspection for the last week of December. According to personnel interviewed, the primary inspector was on vacation, and the substitute inspector used an obsolete form to document the inspection. 18

19. Step 5 – Write the Deficiency Statement & Discussion (cont.) Example B:Calibrations for Instrument AB are not recorded in logbook IAB. According to the site procedure, the calibration information is recorded separately and must be attached to the logbook. Instead, the calibration sheets are bound separately, and no logbook references to the bound sheets were found. 19

20. Step 5 – Write the Deficiency Statement & Discussion (cont.) Example C:Tank sampling is not described in a procedure. Tank sampling is performed to determine compliance with discharge requirements. Procedure AA directs staff to sample the tank, but it does not describe the steps necessary to ensure compliance with the discharge requirements. According to personnel interviewed, the necessary steps are not described in any other procedure. 20

21. Step 5 – Write the Deficiency Statement & Discussion (cont.) Examples D:Employees have not performed continuing demonstrations of capability. The audit team’s document review identified training records for two employees that show overdue annual continuing demonstrations of capability. The supervisor verified that the DOCs have not been performed. E:Device A is not used to perform exposure monitoring. According to site documents, as of January 2013, Device B has been used for exposure monitoring. However, procedure RAD-100 has not been updated to reflect the change to Device B. 21

22. Step 6 – Write the Finding Statement • Write the Finding Statement last, adding the appropriate Priority and References. • The Finding Statement can be a generalization of the Deficiency Statement. 22

23. Step 6 – Write the Finding Statement (cont.) Examples A:Storage area inspections are not documented in accordance with requirements. B:Instrument calibrations are not properly recorded. 23

24. Step 6 – Write the Finding Statement (cont.) Examples C:The facility’s procedures do not describe all of the required activities. D:Annual continuing documentations of capability are not performed. E:Exposure monitoring is not performed in accordance with requirements. 24

25. Step 6 – Write the Finding Statement (cont.) • Add the References (separate them with semicolons). • Ensure that all aspects of the applicable part of the requirement are covered in the references. You can include “overarching” references if necessary for clarity. • In general, cite the references: • Highest to lowest • Least specific to most specific 25

26. Tips • Keep the Finding on point! • Try to write the Finding Statement without using an acronym. If you find that you can’t do that, you may be getting too specific. • If you need to include specific instances of the deficiency so that the facility will better understand it, refer to them as examples of the deficiency (i.e., as shown by the following examples . . .). 26

27. Tips (cont.) • Remember that findings must be “closeable.” • If the facility uses a term or phrase that is different from the requirement but means the same thing, clarify it in a sentence that ties the words in the requirement to the words used by the facility. • Help the reader make the connection between the two terms/phrases. 27

28. Summary – The Six Steps • Step 1 – Identify the deficiency. • Step 2 – Identify the requirement. • Step 3 – Pull the string and notify the Lead Auditor. • Step 4 – Write the Requirement Statement. • Step 5 – Write the Deficiency Statement and associated Discussion. • Step 6 – Write the Finding Statement and add the Priority and References. 28

29. Example A Deficiency A-1:Didn’t find documentation of a weekly inspection of a storage area for the last week of December. Requirement A-2:Permit requires that inspections of storage areas be performed weekly and documented on Form X. Requirement Statement A-3: Storage area inspections must be documented on Form X. Deficiency Statement & Discussion A-4: Form X was not used to document the storage area inspection for the last week of December. According to personnel interviewed, the primary inspector was on vacation, and the substitute inspector used an obsolete form to document the inspection. Finding Statement A-5:Storage area inspections are not documented in accordance with requirements. 29

30. Example B Deficiency B-1:Didn’t find a required calibration record. Requirement B-2:The QSAS requires instruments to be calibrated and the calibration documented. Procedure CAL-100 requires documentation in the logbook specific to the instrument. Requirement Statement B-3:Calibrations for Instrument AB must be recorded in Logbook IAB. Deficiency Statement & Discussion B-4:Calibrations for Instrument AB are not recorded in logbook IAB. According to the site procedure, the calibration information is recorded separately and must be attached to the logbook. Instead, the calibration sheets are bound separately, and no logbook references to the bound sheets were found. Finding Statement B-5:Instrument calibrations are not properly recorded. 30

31. Example C Deficiency C-1:Observed a sampling step that was not addressed in a procedure. Requirement C-2:The QSAS and NQA-1 require all activities to be described in a procedure. Requirement Statement C-3:Tank sampling must be described in a procedure. Deficiency Statement & Discussion C-4:Tank sampling is not described in a procedure. Tank sampling is performed to determine compliance with discharge requirements. Procedure AA directs staff to sample the tank, but it does not describe the steps necessary to ensure compliance with the discharge requirements. According to personnel interviewed, the necessary steps are not described in any other procedure. Finding Statement C-5: The facility’s procedures do not describe all of the required activities. 31

32. Example D Deficiency D-1:The training records for two employees show overdue annual continuing demonstrations of capability. Requirement D-2:Procedure TWY-101 requires employees to perform annual continuing demonstrations of capability. Requirement Statement D-3:Employees must perform annual continuing demonstrations of capability. Deficiency Statement & Discussion D-4:Employees have not performed annual continuing demonstrations of capability. The audit team’s document review identified two examples of training records that show overdue annual continuing demonstrations of capability. The supervisor verified that the DOCs have not been performed. Finding Statement D-5:Annual continuing documentations of capability are not performed. 32

33. Example E Deficiency E-1: Observed personnel exposure monitoring using wrong device. Requirement E-2:RML requires exposure monitoring to be performed in accordance with procedures submitted with the license application. Procedure RAD-100 requires use of Device A for exposure monitoring. Requirement Statement E-3: Device A must be used to perform exposure monitoring. Deficiency Statement & Discussion E-4:Device A is not used to perform exposure monitoring. According to site documents, as of January 2013, Device B has been used for exposure monitoring. However, procedure RAD‑100 has not been updated to reflect the change to Device B. Finding Statement E-5:Exposure monitoring is not performed in accordance with requirements. 33

34. Lead Auditors • Your job as lead keeps going after you leave the facility. • While the facility is conducting its factual accuracy review, this is your opportunity to review the whole report. • Pay particular attention to all the findings and ask yourself the same questions the auditor should ask. • If you spot something that’s not quite right about a finding or if you think the wording needs clarification, contact the auditor. 34

35. Changes to the CAP Template & CAP Closure Verification • DOECAP has revised the CAP form to make it clear that facilities must provide: • Corrective actions to fix the deficiency AND • Actions to prevent recurrence. • When the audit team verifies actions to close the finding, they must verify both. • If both actions cannot be verified as adequately completed, the finding remains open. 35

36. Official Use Only (OUO) Audit Reports – Report Templates – Emails • Audit Reports are OUO. • Audit Report Templates sent out by the Operations Team contain OUO text from the previous year’s audit report, and they are also OUO. • Comment Emails on audit reports or CAPs are OUO and the Subject Line must identify if the text or attachment is OUO. 36

37. Questions? 37

38. Observations Format is less prescriptive. Tips • Try to get to the point right away. • If it sounds like a requirement is involved, clarify. • Isolated or one-time noncompliances are findings. 38

39. Observations (continued) • Avoid any implication of a suggestion. No “should” or “may want to consider” or “it would be prudent to” - type phrases. • The observation must pass the “so what?” test. Why is it an issue? • State what negative consequences are possible for the facility if it keeps doing what it’s doing now. 39

40. Observations Examples of Negative Consequences • Failure to keep all copies of the emergency contacts list up to date could lead to the wrong person being called in an emergency. • Failure to trend MAPEP data could result in reduced data quality and contribute to failing proficiency test results. 40

41. Questions? 41

42. Previous FindingsSome Do’s and Don’ts 42

43. Closing Previous Findings • Do answer these four questions: • How do you know? • What changed? • Is the change adequate to fix the deficiency? • Did the facility complete the action identified to prevent recurrence? 43

44. Closing Previous Findings • How do you know? Include a phrase that tells the reader what you did to verify closure. • Reviewed the revised procedure • Observed the activity • Reviewed the records (e.g., training records, shipping manifests) • Interviewed personnel 44

45. Closing Previous Findings • What changed? • Briefly state what the facility did to fix the problem (e.g., added something to the procedure or took something out, provided training, revised a form). • Is the change adequate to fix the deficiency? • Continue by stating what you determined from your review/observation/interview (i.e., whether or not the action actually fixed the problem). If the problem has recurred, your answer is “no.” 45

46. Closing Previous Findings • Did the facility complete the action identified to prevent recurrence? • The facility won’t always need to identify an additional action to prevent recurrence, but if the CAP lists one, you must verify that it has been completed. 46

47. Closing Previous Findings Some Do’s and Don’ts • Don’tpaste in the previous requirement and deficiency text. Paraphrase or summarize the issue if necessary for clarity. • Dobe clear about what is old information and what is new information. • Do try to make it obvious from your choice of words whether or not the finding is closed or remains open. Avoid boilerplate conclusion statements. 47

48. Closing Previous Findings Summary • Answer the 4 Questions • How do you know? • What changed? • Is the change adequate to fix the deficiency? • Did the facility complete the action identified to prevent recurrence? • Get right to the point. 48

49. Open Findings • Summarize the issue. • Clarify why it is still open. • If the issue was partially addressed, provide sufficient explanation for future closure. 49

50. Questions? 50