Audits Overview: NPAP, PEP, Annual PE, Flow Rate Audits

1. Audits Overview: NPAP, PEP, Annual PE, Flow Rate Audits The Measuring Sticks of Air Monitoring Guest Speaker: Jeremy Howe Little River Band Of Ottawa Indians

2. Overall Course Overview: • April 7: QA Systems, EPA definitions, PQAOs and common sense – Mike Papp • April 14: Routine Quality Control and Data Management (1-pt QC, flow rate, and instrument stability checks) – Travis Maki • April 21: Audits Overview (NPAP, PEP, Annual PE, Flow Rate Audits) – Jeremy Howe • April 28: Calculating Bias and Precision and AQS reports – Bill Frietsche • May 5: 40 CFR 58 App. A- Gaseous Pollutants – Glenn Gehring • May 12: 40 CFR 58 App. A- Ozone – Brenda Jarrell • May 19: 40 CFR 58 App. A- PM filter and continuous methods –Brandy Toft

3. Audit definition: • Audit – “a systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives” • Independent means using a device other than the one you use routinely at a minimum, and in some cases a completely independent system of instrument and operator

4. All Audits (in some cases called “assessments”): • Basically compare • What is actually being done in the field and the office • Against what is stated in the QAPP and SOPs • Are done by you or someone outside your office, depending on the type of audit

5. Describe in your QAPP • Number and types of assessments • People or organizations doing the assessments • Schedule • Criteria for assessments (we will compare what we do against the SOP or QAPP) • Reporting and responsibility for follow-up (e.g., we will revise the SOP in pen for now, and retype it with changes within a month)

6. Responsibility for Follow-Up and Verification of Corrective Action Program: CLOSE THE LOOP (fix problems and take action to make sure they do not happen again) And make sure it is documented

7. 4 categories of audits: • 1. Six-month flow rate audits you make with non-routine standards (PM) • 2. Annual Performance Evaluations (gas) • 3. NPAP and PEP audits conducted by certified auditors • 4. Technical Systems Audits conducted by EPA regions (every 3 years) • To see how many/often of the 1st three types you need for NAAQS-comparable data, see Calculations for Tribes Interested in Joining a Tribal PQAO - 2011-02-08A.xls

8. QC Checks (and audits):

9. Who can do these, and with what?

10. First type: PM flow rate audits (counts as an audit, for bias, because it is not the stnd used for precision FR checks):

11. A PM2.5 example, using the spreadsheet:

12. This yields, in the QC Checks tab:

13. And in the Audits tab:

14. 1st type, cont.: PM Flow Rate Audits: • Discussed in more detail in PM webinar • Rely on owner’s manuals • EPA guidance available: • 2.12 "Monitoring PM2.5 in Ambient Air Using Designated Reference or Class I Equivalent Methods" 1998 http://www.epa.gov/ttn/amtic/files/ambient/pm25/qa/m212covd.pdf • See docs in course webpage and AMTIC and CARB SOPs

15. 1st type, cont.: Semi-Annual Flow Rate Audits (PM): • check the operational flow rate of the analyzer, before any flow rate adjustment • Randomize the time and day • Use a flow rate transfer standard certified to be traceable to an authoritative volume or other applicable standard (within 2% of NIST –Traceable Standard Part 50, App L Sec 9.2) • Reported to AQS as RA (bias) because it is made with a non-routine standard (the ones with your routine stnd are precision (RP) • % diff =[ (meas-audit)/audit ]*100

16. 2nd type of audits: Annual Performance Evaluations (gas) • 3.2.2  Annual performance evaluation for SO2, NO2, O3, or CO • …The gas standards and equipment used for evaluations must not be the same as the standards and equipment used for calibration or calibration span adjustments…the auditor should not be the operator or analyst who conducts the routine monitoring, calibration, and analysis.

17. 2nd type of audits: Annual Performance Evaluations, gas, cont.: • 40 CFR 58 App A sec 3.2.2: • If there are fewer than four analyzers for a pollutant within a PQAO, P.E. annually (preferably in different quarters) • The evaluation should be conducted by a trained experienced technician other than the routine site operator • Current CFR guidance will change soon

18. A gas analyzers example, 3 at 1 site:

19. This yields, in the Checks tab:

20. And in the Audits tab, this yields 1 NPAP and 3 Annual PEs:

21. 2nd type of audits: Annual Performance Evaluations, gas, cont.: • use…audit gas from at least three consecutive audit levels. The audit levels selected should represent or bracket 80 percent of ambient concentrations measured by the analyzer being evaluated:

22. 2nd type of audits: Annual Performance Evaluations, gas, cont.: • New guidance for audit levels: • Ten levels, the lowest as low as 4-6 ppb for O3 • Consecutive audit levels no longer required • See memo: • http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/EPA%20guidance%20and%20other%20reference%20material/TechMemoforPEAuditLevels.pdf

23. 2nd type of audits: Annual Performance Evaluations, gas, cont.: • 3.2.2…Report both the audit and measured concentration • The % differences (di ) between these concentrations are used to assess the quality of the monitoring data, and to confirm your biweekly QC check results:

24. 2nd type of audits: Annual Performance Evaluations, gas, cont.: • Allow…the audit gas to pass through all filters, scrubbers, conditioners, and other sample inlet components used during normal ambient sampling and as much of the ambient air inlet system as is practicable • < 15 % for each audit concentration, and • EPA uses these (RA) values you report to AQS to generate probability limits for your bias

25. 3rd type of audits: NPAP and PEP audits conducted by certified auditors 2.4  National Performance Evaluation Programs. ….the consent of the monitoring organization for EPA to apply an appropriate portion of the grant funds, which EPA would otherwise award to the monitoring organization for monitoring activities, will be deemed by EPA to meet this requirement. For clarification and to participate, monitoring organizations should contact either the appropriate EPA Regional Quality Assurance (QA) Coordinator…

26. 3rd type, cont.: NPAP and PEP audits conducted by certified auditors (cont.) • PEP- $2000/audit • 5 audits/yr for PQAO with < 5 sites = 10K/year • All sites audited at least once every 6 years • NPAP- $2200/audit • Each site audited at least once every 5 yrs • At least 1 NPAP audit per PQAO network that includes even 1 gas analyzer • Would need 10 sites for 2 audits a year • Each NPAP audit covers all gas analyzers at a site • These (rough) costs are for EPA contractors doing the audits

27. 3rd type, cont.: PEP and NPAP audits • Auditors (including tribal folks) can take training and exam • Equipment must be approved/provided as audit-specific equipment • SOPs available on AMTIC, http://www.epa.gov/ttn/amtic/npepqa.html

28. NPAP Audits (3rd type, for gas) • Separate from Cradle to Grave • For Gaseous Pollutants • O3 + CO + SO2 + NOX

29. PEP Audits (3rd type, for PM2.5) • Separate from Cradle to Grave • For Filter Based Pollutants • PM 2.5 + Lead PEP Audit Collocation PEP

30. 4th type of Audit: Technical Systems Audits conducted by EPA regions (every 3 years) • 2.5  Technical Systems Audit Program. Technical systems audits of each ambient air monitoring organization shall be conducted at least every 3 years by your EPA regional office • Review of siting, documentation, procedures’ implementation • Assessment of whether the “QA function is independent” • Checklists that the region will use are available as QA Handbook Volume II, Appendix H

31. 4th type, cont.: EPA Technical Systems Audits • At least 3 months notice, with written audit plan • Checklist sent ahead of time, see NAAM Technical Systems Audit Form in: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/EPA%20guidance%20and%20other%20reference%20material/

32. How are audit results used? • All audits are “friendly” and intended to improve the quality of the data • Findings will always include insufficient documentation • Site visits are included • Some AQS data are pulled, then records of all the QC and original data are looked for in office • Important for small organizations: the “independency of the QA function”

33. “Independent” QA Function: • CFR App. A: 2.2  …a quality assurance management function…includes strategic planning, allocation of resources and …assessing and reporting… • must have sufficient technical expertise and management authority to conduct independent oversight… • should be organizationally independent of environmental data generation activities (e.g., NOT the site operator or their manager) • Ideal QA manager is described at: http://www.epa.gov/ttn/amtic/files/ambient/monitorstrat/qamroles2.pdf

34. Audits Conclusions: • Adds necessary quantitative and qualitative (TSAs) yardsticks for data defensibility • Adds value by exchanging information • Course website: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/ • Consider joining the TAMS Steering Committee to help develop national tribal training to address these complicated issues (email us) • Jeremy Howe: jhowe@lrboi.com • Melinda.ronca-battista@nau.edu • Christopher.lee@nau.edu