Quality Assurance Protocol: WRAP RMC Emissions Modeling with SMOKE

1. Quality Assurance Protocol:WRAP RMC Emissions Modeling with SMOKE Zac Adelman Carolina Environmental Program UNC Chapel Hill Presented October 8, 2003 at UC-Riverside/WRAP RMC

2. Outline • Introduction • QA Tenets • SMOKE Quality Assurance Framework • Implementing the framework • Specific QA Procedures • QA Products

3. Introduction • QA protocol provides an explicit framework for SMOKE modeling procedures • Defines expectations for data quality • Allows pre-planning for additional effort • Multiple levels of QA • Outlines modeling, documentation, and archival requirements for thorough QA • Facilitates reproducibility of SMOKE results

4. QA Tenets • Organization • Planning • Control • Clean • Question • Document • Vigilance

5. SMOKE QA Framework • Implementation: Three-modeler approach • QA Classifications • Required Information • QA Procedures • Templates and Worksheets • QA Products

6. SMOKE QA Framework • Three-modeler approach • Production modeler* • Data collection, running SMOKE, maintaining scripts, creating QA reports, archival, delivering SMOKE output, QA documentation • QA manager • Enforcing adherence to QA protocol, leads QA documentation effort • Principal investigator • Primary technical advisor, maintains project schedule, final report writing

7. Three-Modeler QA Approach

8. QA Classifications • Model QA: Accuracy assurance and problem identification • System QA: Software and data tracking • Documentation

9. Required Information • Program logs • SMOKE QA reports • Run scripts • Assigns files • Inventory summaries • QA documents and worksheets

10. Model QA • Procedures for determining if SMOKE is working correctly and producing expected results • Standardized approach for scrutinizing emissions model I/O establishes reference points for data quality

11. Model QA Components • Confirm SMOKE configuration • Checking inventories and ancillary input files • Confirm SMOKE execution and check logs • Confirm that the inventories processed correctly • Confirm the information in the SMOKE QA reports • Examine indicators for model performance

12. System QA • QA process for facilitating reproducibility • Accounting system for models and data • Begins with installation/benchmarking • Tracks all updates/changes to software and data • Completes with archival and recording

13. System QA Components • Archiving SMOKE source and scripts with CVS • Installation freeze points • Tracking SMOKE data with QA worksheets

14. Documentation • Record results, progress, and key features of Model and System QA • Central to software and data tracking procedures • Tangible record of SMOKE modeling process and how to reproduce the results • Includes a final report summarizing all modeling and QA results

15. Documentation Components • Recording SMOKE configuration options • Recording SMOKE input/output data • Documentation of the QA process • Final report • Automated project management tool

16. QA Protocol Specifics • Step-by-step QA procedures • SMOKE program-specific QA checks • SMOKE Troubleshooting • Procedures for identifying, fixing, and documenting emissions problems • QA Summaries • Procedures for reporting QA process, documenting troubleshooting, and compiling a final report

17. Step-by-Step QA Procedures • Performed by the production modeler, confirmed by the QA manager and PI • Specify options, SMOKE flags, and features to look for in the scripts, logs, and outputs from all SMOKE processors • Coincide with a QA checklist for tracking the progress of these procedures

18. Example QA Steps • Inventory Import (Smkinven) • Inventory data – cross reference the Smkinven log with the completed SMOKE inventory file worksheet from Appendix C to confirm that the correct inventory files are being modeled • Annual, daily, or hourly emissions – check that the SMOKE setting for importing daily/hourly files or annual/daily inventory records is consistent with the current inventory

19. SMOKE QA Checklist • Detailed explanation of the steps in the checklist presented in the body of the protocol • Each step requires modelers initials who checked the step • Steps contain comment fields to answer questions associated with the step • Final checklist requires modeler, QA manager, and PI to sign for certification of completion

20. QA Products • Completed Emissions QA checklist • Completed QA templates and worksheets • SMOKEv2.0 settings • Inventory file log • Ancillary input file log • SMOKEv2.0 execution log • SMOKE QA reports and graphics • Final report

21. Final Thoughts • QA protocol development is an evolving process • This protocol presents an experiment in micromanaging emissions modeling • Experience has shown that in the reality of emissions modeling there are no “unexpected problems”; expect and prepare!!!