Lora S. Johnson, Director of Quality Assurance Gary J. Foley, Laboratory Director

1. An Evaluation of the EPA Quality Assurance Annual Report and Work Plan from the Perspective of a Traditional Cost of Quality Model Lora S. Johnson, Director of Quality Assurance Gary J. Foley, Laboratory Director USEPA, National Exposure Research Laboratory 18th Annual National Conference on Managing Quality Systems for Environmental Programs Cincinnati, Ohio April 16, 1999 NERL

2. OVERVIEW REPORTING ISSUES COST OF QUALITY CATEGORIES ELEMENTS OF THE QUALITY ASSURANCE ANNUAL REPORT AND WORKPLAN (QAARWP) QAARWP VS. COST OF QUALITY CATEGORIES ORD APPROACH FOR FY98/99 A POTENTIAL SOURCE OF COST DATA IDEAS FOR REFINING THE QAARWP NERL

3. REPORTING ISSUES Managers need information so that appropriate resources can be allocated to quality assurance activities. What information is needed? How should it be measured? How should it be collected? How should it be represented? The Agency uses the Quality Assurance Annual Report and Work Plan (QAARWP) to provide this information. How does the EPA approach compare to a traditional private sector assessment of the “cost of quality?” NERL

4. COST OF QUALITY CATEGORIES PREVENTION The costs of all activities specifically designed to prevent poor quality in products and services. Quality assurance project plans Training in quality QA review APPRAISAL The costs associated with measuring, evaluating, or auditing products or services to assure conformance to quality standards and performance requirements. Calibration Performance evaluation samples Technical system audits NERL

5. COST OF QUALITY CATEGORIES FAILURE The costs resulting from products or services not conforming to requirements or customer/ user needs. INTERNAL FAILURE Costs that occur prior to delivery Reworking publications Too many signatures Failed Experiments EXTERNAL FAILURE Costs that occur after delivery ??? NERL

6. ELEMENTS OF THE QAARWP FROM THE EPA QUALITY MANUAL, 7/98 QA Annual Report Quality Management Resources Total FTE’s Federal FTE’s - QA Management Federal FTE’s - QA Technical Support Contract FTE’s - QA Direct Support Contract FTE’s - QA Nontechnical Support Contract $ $ for Travel-Oversight Activities $ for Travel/Training about QA Resources adequate? Training Method used to assess training needs Course names and suppliers Attendees Course goals met? NERL

7. ELEMENTS OF THE QAARWP FROM THE EPA QUALITY MANUAL, 7/98 QA Annual Report (continued) Management Accomplishments Innovative Practices Technical Assessments Technical Assistance New/revised QA Guidance Publications Awards Management Assessments MSR’s - Internal MSR’s - External Work Plan Quality Management Resources (same elements as above) Activities NERL

8. QAARWP VS. COST OF QUALITY CATEGORIES Quality Management Resources A, P Training P Management Accomplishments A, P Management Assessments A A = Appraisal CostP = Prevention Cost NERL

9. ORD APPROACH FOR FY98/99 GOAL: ACCOUNT FOR RESOURCES IN A SIMILAR WAY TO PROMOTE COMPARABILITY IN THE ORD RESOURCE SUMMARY TABLE METHOD: DIRECTORS OF QA FOR NERL, NHEERL, NRMRL MET WITH QAD REPRESENTATIVE VIA TELECONFERENCE GROUP AGREED TO ACCOUNTING RULES FOR ELEMENTS IN QUALITY MANAGEMENT RESOURCES CATEGORY EXAMPLE: Quality Management Resources = Sum of the Division QA Managers and the Director’s of QA i.e. staff listed on EPA’s QA Community list, adjusted for percentage of time spent on the QA function NERL

10. A POTENTIAL SOURCE OF COST DATA FEDERAL GOVERNMENT CUSTOMER SERVICE INITIATIVES Vice-president’s “National Performance Review” seeks to “build a government that works better and costs less.” Customer service standards are developed across the Federal government. Congress passes the Government Performance and Results Act of 1993 that seeks to improve Federal program effectiveness and public accountability by promoting a new focus on results, service quality, and customer satisfaction. ORD’s products are knowledge based therefore “failures” are hard to measure. Customer surveys are a potential tool for measuring both success and failure. Including customer survey data in the QAARWP would close the loop on cost of quality items. NERL

11. IDEAS FOR REFINING THE QAARWP Better Definition of Terms: Need to compare apples to apples Include customer survey data: Need some measure of “cost of failure” Examine use of the QAARWP: Is information helpful for making management decisions? Index QA resources to some measure of productivity: QA FTEs/FTEs QA FTEs/$ QA FTEs/ # APMs Present data graphically NERL

12. Customer Satisfaction QA Index Time Customer Satisfaction QA Index Time Customer Satisfaction QA Index Time HYPOTHETICAL SCENARIOS COMMENTS QA Effect Positive Why are we bothering? QA Hurts NERL

13. CONCLUSION The QAARWP is a significant element of the EPA quality system. To make it a more useful management tool, some estimate of “failure” needs to be included. Recent efforts in the area of customer satisfaction may provide a surrogate measure of external failure at least to the EPA R&D laboratory community. NERL