Presentation Transcript

1. Precision, Bias, and Total Error (Accuracy)

   Guest Speaker: Bill Frietsche US EPA
2. Overall Course Overview: April 7: QA Systems, EPA definitions, PQAOs and common sense – Mike Papp April 14: Routine Quality Control and Data Management (1-pt QC, flow rate, and instrument stability checks) – Travis Maki April 21: Audits Overview (NPAP, PEP, Annual PE, Flow Rate Audits) – Jeremy Howe April 28: Calculating Bias and Precision and AQS reports – Bill Frietsche May 5: 40 CFR 58 App. A- Gaseous Pollutants – Glenn Gehring May 12: 40 CFR 58 App. A- Ozone – Brenda Jarrell May 19: 40 CFR 58 App. A- PM filter and continuous methods –Brandy Toft
3. Error …the difference between your answer and the “truth” Two components of total error (accuracy): Bias (jump) Precision (wiggle)
4. Precision Error Some imprecision is unavoidable Sometimes up, sometimes down–“random” Difference divided by best estimate of the truth “truth” is: For gas QC checks: known conc. For PM flow rate: audit (known) FR For PM2.5 collocated: their average
5. 1st way of assessing precision: Repeated measurement of the same thing di = the difference between a known value and your value (flow rate, conc., voltage, that should remain the same) Sequence of check (date, time, check number)
6. 2nd way of assessing precision: Pairs of simultaneous measurements (collocated instruments) di(difference between collocated values) date
7. Bias = calibration error: Can be sudden or due to slow drift over time: di = the difference between the true value (audit value) and the analyzer date
8. Accuracy = total error Includes both bias and precision At any one time, an audit value could be close to the analyzer value due to precision errors in both Or, audit value could be far from analyzer value due to precision errors adding Audits estimate Accuracy EPA uses audit results to verify that ongoing QC checks do represent total error
9. Accuracy = total error: Auditor’s see only one day, while your QC checks see overall big picture
10. Precision and total error both are estimated from the QC checks: 1-pt QC checks no longer called precision checks, because the results are used (by YOU) to calculate both precision and to estimate total error Each check <= 7% is the CRITICAL criteria for each set of data since last passing check The audits basically verify your precision and bias that have been calculated all along from your QC checks (concentration and flow rate, for PM)
11. QC Checks are used to estimate precision*: *and total error, more on this later….
12. Why is tracking precision important? Estimates your overall system uncertainty, if you were perfectly calibrated Changes over time can point to operator error, lab error, poorly written procedures, equipment/standards going bad Changes can be fixed sooner rather than at audit EPA compares precision by site and agency
13. Control charts: Useful to track precision (or bias) over time See this example (and you fill in your values) at: Another example in the DASC tool: di
14. Control chart in PBDASC tool: The DASC tool automatically writes your values from the columns into the chart: Measured value is your analyzer Audit value is the known di You may want to add dates to the x-axis, or use my example if you want to set up control charts in your program
15. EPA estimates precision: In their annual QA reports, from your RP transactions in AQS: http://www.epa.gov/ttnamti1/qareport.html http://www.epa.gov/ttnamti1/anlqa.html di
16. Invalidation criteria for NAAQS: Recommendations in redbook say: Critical criteria invalidate every hour that is not met Operational criteria means something probably wrong, go check it Systematic criteria mean as a set (day? year?) data is not usable for NAAQS, but individual hours or more may be valid
17. Critical QC invalidation criteria for gas: (NO2 and SO2 are the same)
18. Critical invalidation criteria for PM:
19. Operational precision invalidation criteria for PM:
20. Total Error Keep the bias component minimized by calibrating and verifying your equipment against a standard Keep precision component low by consistency You can work to keep bias down, while precision is often out of your control below a certain limit EPA calculates from your RA transactions
21. Total error requirements for NAAQS are all operational criteria (investigate, but not necessarily invalid):
22. Bias, precision, and total error: Bias: systematic difference, or “jump” Precision: random error, or “wiggle” Accuracy = total error, a combination of both Simple temp bath illustrations at: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/AllDownloadableMovies/
23. How do I calculate precision and total error? DASC (Data Assessment Statistical Calculator) http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/DASC%20EPA%20Prec%20and%20Bias%20Calculator/ AQS: Data Quality Indicators Report (AMP255)
24. Before we cut to Bill: Minimize bias by regular calibrations both accuracy (total error) and precision are first estimated by di on an ongoing basis, by you Audits verify total error estimates Course website: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/ Our emails: Bill Frietsche: frietsche.bill@epamail.epa.gov Melinda.ronca-battista@nau.edu Christopher.lee@nau.edu

25. AMP255 and other AQS QA tools

26. Appendix A to Part 58 Table A-2 of App A AQS TTN web site AQSP&A Spreadsheet Discoverer AMTIC web site AQS Helpline AQS AMP255 Data Quality Indicator Report
27. Appendix A to Part 58 Regulations that define QA reporting requirements for criteria pollutants Define assessments for each criteria pollutant 1 Pt QC check for gases Annual performance evaluation for gases Flow rate verification for particulate matter Semi-annual flow rate audit for particulate matter Collocated sampling requirements for particulate matter Pb audit strips for laboratory analysis QA Performance Evaluation Program for PM fine, PM coarse, and Pb Formerly called precision and accuracy data – still use these terms on AQS transactions
28. This document available at: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/session%204-prec%20&%20bias/
29. AQS TTN web site http://www.epa.gov/ttn/airs/airsaqs/ AQSP&A Spreadsheet http://www.epa.gov/ttn/airs/airsaqs/padata/ AQS Discoverer http://www.epa.gov/ttn/airs/airsaqs/aqsdiscover/ AMTIC web site http://www.epa.gov/ttn/amtic/ AQS Help Line EPACALLCENTER@epa.gov 1-866-411-4EPA (4372)
30. Example run of AMP255 shown onscreen
31. AQS Precision and Accuracy Conclusions: All QC data helps EPA balance your costs of QC with needed information to protect health Questions to AQS are welcomed EPA committed to improving user friendliness Course website: http://itep68.itep.nau.edu/itep_downloads/QA101_Resources/ Our emails: Bill Frietsche: frietsche.bill@epamail.epa.gov Melinda.ronca-battista@nau.edu Christopher.lee@nau.edu