Quality Assurance Program Presenter: Erin Mustain

1. Quality Assurance ProgramPresenter: Erin Mustain

2. Recommendation 5Benchmarks • Data quality issues have been categorized and quantified. • A detailed plan exists for addressing sources of continuing errors and correcting historical errors. • The plan has been validated with representative data samples • Substantive progress has been made toward correcting major categories of errors. • The Steering Committee agrees that progress is being made and that there is a high probability that existing data problems will be resolved.

3. Progress • Institutionalized Quality Assurance • Business Rules • Error chart • Steering committee and User groups collaboration • Eliminated system-generated violations for paper tracking SMRs 3

4. Total Study Error Data Population Data Generation Training Migration Manual Data Entry System Limitations SMRs Manual Non-numeric data (ND, QND, etc.) not handled by eSMR Training Manuals don’t follow Business Rules Errors in data entry form Field not populated Business Rules not followed Lab errors System can’t handle unique orders (several facilities under one permit) Field auto-populated incorrectly Typos Sampling Errors Calculation errors Instruction not consistent Difficulty Data Mining No place to store data (enrollee history) Field not appearing Intentional manual errors Lack of Training Selecting the wrong link Doesn’t enforce all of the Business Rules Data doesn’t follow business rules Duplicate entry Can’t easily delete records Data entered into SWIM incorrectly System generated duplicates (SMARTS)

5. Current vs. Historical Data 5

6. What decisions will be made with this data? SOPs QAP Audits Business rules Data Validation CIWQS QA Program Data Cleanup DQOs Data Verification Training Communication Corrective Action QA Reports Integration with State Water Board QMP 6

7. Plan and Procedures • Quality Assurance Plan • Scope • Roles and responsibilities • Data quality indicators • Quality objectives • Assurance activities • Problem reporting and corrective action • Audits • Migration and future projects 7

8. Plan and Procedures • Standard Operating Procedures • Data Cleanup • Training • Document Management • Corrective Action • Quality Assurance Reports • Audit • End-user-layer enhancements and testing • Database enhancement prioritization • Report prioritization

9. Plan and Procedures • DIT Standard Procedures Document • Maintenance & Documentation Requirements • System Environment • Data model, database, data integration, & maintenance • Application source code integration & maintenance • Application and database source code & scripts repository CVS Database tools & scripts standards • Issue routing • Maintenance implementation

10. External Audit Policies and procedures for data entry Onsite audit of Regional Boards and State Board programs Security, performance, and policies and procedures Onsite audit of Division of Information Technology Data audit using stratified random design approach Accuracy of records 10

11. Next Steps • Quantifying the data quality issues – Audit • Correcting historical data – recommendations to management after audit results • Validate the QAP and SOPs with representative data samples • Implement training program • Continuing process improvement at all levels (QA Program is not static) • Establish a mechanism for communication between QA Program and panel 11