Framework for Providing Quality Assurance Guidance for Computer-Based Software Models used in Research Projects and Regu

1. Eric S. Hall Surender Kaushik Human Exposure and Atmospheric Sciences Division National Exposure Research Laboratory Framework for Providing Quality Assurance Guidance for Computer-Based Software Models used in Research Projects and Regulatory Support Activities

2. Introduction • Standard Quality Assurance/Quality Control Processes • Applied at different points in the product life-cycle • Design (specifications: describe what is required) • Manufacturing (statistical process control) • Maintenance (mean time to repair) • Place emphasis on numerical analysis • Are intuitively understood in familiar contexts • Measurement precision (10 mL + 0.25 mL) • Statistical analysis (mean, variance, R2, %-iles) • Provide measurable criterion for assessing ‘good’ • Hard to apply in qualitative assessment contexts

3. Introduction • EPA relies on computer programs/software/models • Conduct normal business (administration) • Support regulatory decision-making activities • Develop/implement research programs • Some computer programs • Purchased in the normal marketplace (off-the-shelf) • Office use (word processor, spreadsheet, etc.) • Special use (graphics, statistical analysis, etc.) • Made specifically for EPA use (custom-’built’) • Designed/developed by EPA employees • Created by non-EPAsources (OUR FOCUS)

4. Computer Models In Research EPA’s National Exposure Research Laboratory (NERL) Conducts research to characterize exposures Objective of exposure research is to identify and characterize environmental stressors (pollutants) their effects on receptors (people) NERL researchers seek to understand: processes affecting transmission of pollutants from their sources their transformation in the environment the interaction of pollutants with individuals and populations NERL’s exposure research: conducted within a source-to-outcome continuum provides a framework to evaluate the critical linkages between pollution sources associated human health impacts 3

5. NERL Source-To-Outcome Framework 4

6. NERL Computer Models NERL develops models: Examples of NERL-developed models source apportionment/receptor models [PMF/UNMIX/APTR] air quality models [CMAQ] and [AERMOD] Bayesian environmental concentration (‘fusion’) models [HBM] exposure models [SHEDS] and databases [CHAD] Land-use regression [LUR] dose models [ERDEM] Focus is on air pollutants within NERL source-to-outcome framework to conduct exposure research to summarize our knowledge of exposure processes mathematically quantify/predict concentrations of chemical pollutants, biological and physical conditions, exposures, and dose use of models is central to EPA’s regulatory decision making Key use of these models is in ‘accountability’ research to determine the impact of regulatory changes on pollution levels on human health impacts 5

7. XXXXXXXXXXXXXXXX Mechanistic Air Quality Models Air Quality: Exposure Framework Regional Scale, Population Level CMAQ SHEDS ERDEM • Exposure: • Population • Demographic Info • Time-activity & Commuting • Housing factors AQ Model Input • Ambient Concentration • Ozone, PM2.5 species, toxics • Regional Airshed • Spatially gridded (36km,12km, 4km) • Hourly values: months, years • Dose: • Population • Physiological • Activity level • ADME • Effects: • Population • Acute • Chronic • Source/Stressor Characterization • National Emission Inventory (NEI) • “SMOKE” processing • Spatial allocation to county, road, or point source locations • Temporal allocation (hourly to annual) AERMOD • National to Region Scale Concentration: • Fusion of observational data and model output • Ozone, PM2.5 total • Spatially gridded (12km) • Daily values: years Time-Series and Multi-City Epi Studies • Ambient Concentration • Ozone, PM2.5 species, toxics • Regional Airshed • Spatially gridded (36km,12km, 4km) • Hourly values: months, years • Environmental Characterization • Land Use/Landscape • Biogenic emissions (vegetation type, meteorology) • Plume rise from stacks • Meteorology model (WRF, MM5) Improved Air Quality and Health Associations Statistical Models Air Quality: Exposure Framework Urban Scale, Community Level • Transport/Transformation • Dispersion and advection • Atmospheric chemistry • Atmospheric deposition • Spatially gridded (36km, 12km, 4km) • Hourly values: months to years Bayesian & Other • Ambient Concentration • Ozone, PM2.5 species, toxics • Regional Airshed • Spatially gridded (36km,12km, 4km) • Hourly values: months, years • Urban-Scale Concentration • Toxics, other? • Local scale Airshed • Spatially gridded (? km) • Hourly Values: months, years? Intra-urban and Cohort Studies Land Use Regression • Air Quality Monitoring Data • Best estimate where measured • Spatially sparse when large concentration gradients exist • Temporally strong (hourly for PM2.5/ozone) to weak (1 in 3 days) • Monitoring sites change over time • Ambient Concentration • Ozone, PM2.5 species, toxics • Regional Airshed • Spatially gridded (36km,12km, 4km) • Hourly values: months, years SHEDS ERDEM PMF, UNMIX, APTR • Exposure: • Community • Demographic info • Human activity patterns • Housing characteristics • Dose: • Community • Physiological • Activity level • ADME • Effects: • Community & • Individual • Acute • Chronic • Source/Receptor Models • Ambient Concentrations • Emissions profiles • Met Data 6

8. Example: HBM Model Output - PM2.5 Data 7

9. What is called ‘software’ has three important elements • Development Process (design, build, test, deploy, fix) • Software (source code, executable code, etc.) • Documentation (user manual, design/spec., test, etc.) • To obtain high ‘quality’ software: focus on all 3 elements • Most think software is just actual computer program code • Software = Process + Code + Documents (P.C.D.) • Good {Process + Documents} => Quality Software • What quality guidance does EPA have for software? • EPA QA/G-5M • Focus: developing QAPPs for software/models • But the focus on all 3 elements is not complete! Software/Models

10. Software QA/QC • EPA makes regulatory decisions and implements research programs in support of regulatory decisions • Based on using computer software/models • They are ‘one-of-a kind’ • They can not be purchased ‘off-the-shelf’ • They are developed by non-EPA personnel • The models used to support these activities are developed under extramural vehicles/agreements • Contracts/Work Assignments • Task Orders/Delivery Orders • Cooperative Agreements • Interagency Agreements (IA) • CRADA 9

11. Software QA/QC • People working on these extramural vehicles need a resource for (non-software experts): • Contracting Officers (CO) • Contracting Officer Technical Representatives (COTR) aka WAM or WACOR • Task Order Project Officers (TOPO) • Delivery Order Project Officers (DOPO) • Project Officers (PO): contracts, cooperative agreements, interagency agreements, CRADAs • Organization QA Representatives • QA Managers/Directors • Anyone in EPA requiring ‘custom-made’ software • Guidance must be easy to implement 10

12. Software QA/QC • When EPA requires development of ‘custom’ software models in support of its objectives, the expectations should include the following elements (as a minimum): • Software code that operates asexpected/required • Documentation that characterizes the software • User’s Manual/Operator’s Manual • Test Manual (where required) • Design Document • Etc. • A complete copy of the source code for the software (including all scripts, files, etc., used to build the code) • A copy of the executablesoftware/model code • Definition of EPA rights (software, documents, data) 11

13. Software QA/QC • Sound software QA/QC methodology must ensure that EPA obtains models with these characteristics: • Consistent: how the model operates (all conditions) • Reliable: same data should yield the ‘same’ answers • Easy to use: clear user manual, logical process steps • Documented: how its built, how it works, how verified • Implements requirements: does what we want it to • Transparent: understand its use, limits, applicability • The CREDIT Framework for software/model QA/QC • Based on two current elements of extramural vehicles • Scope/Statement of Work/Terms and Conditions • QAPP/QA Guidance 12

14. Software QA/QC • The project officer, COTR, etc., develops the SOW/T&C • SOW/T&C influences how the model is developed and what is delivered to EPA (and should include) • Specific capabilities that model must possess • Code and documents that are to be delivered • Specify QA guidance be incorporated into workplan • Project officer/COTR develops the QAPP/QA Guidance • QAPP/QA Guidance influences • Process to develop/deliver quality software • Software features to facilitate upgrades/revisions • QAPP/QA Guidance must be descriptive (i.e., provide the ‘what’ but the ‘how’ is implemented by the software/model developer) 13

15. Software QA/QC • A resource is available for implementing software QA in our extramural (software/model development) programs • PMRB Software Development QA Guidance Document (PMRB-097.0) • Can be found in NERL/NCCT SOP Module • Lotus Notes based management tool that resides on the EPAHUB2 server • POC: Lora Johnson – NERL/NCCT DQA • Idea developed from PMRB QA Representative • Document length: 6 pages • Contains essential elements for quality software • Serves as a ‘checklist’ for EPA personnel 14

16. Software QA Guidance Document • Purpose: Contractors, grantees, organizations, and individuals developing and delivering software code, software related products, and associated documentation to PMRB for contracts, grants, cooperative agreements, interagency agreements, CRADAs, etc., in support of PMRB’s scientific research activities shall incorporate the provisions of this document into all applicable workplans, QAPPs, QA software plans, scientific/technical guidance, and/or agreements to ensure that delivered software source code, software executable code, user manuals, design documentation, test and validation documents, and all other deliverables adhere to PMRB’s quality standards for software code and related documents and products. The term ‘contractor’ shall refer to any organization or individual engaged in development of software code and related products in support of PMRB’s scientific research activities as a result of a written agreement. 15

17. Software QA Guidance Document • Software Quality Objectives The contractor shall develop software code, software related products and documents in a manner which ensures that the following objectives are met: a) all software source code developed by the contractor shall include identifying and descriptive information (i.e., [list of 14 items]; b) executable software code in all operational and maintenance/test modes shall be developed by the contractor to provide plain-language (English) warning and error messages to users that shall provide clear instructions on how to resume normal software operations; 16

18. Software QA Guidance Document • Software Quality Objectives c) executable software code shall be fully tested/verified by the contractor to ensure that it operates in accordance with all applicable user manuals and test procedures as specified in a workplan, QAPP, or QA software plan; d) interim operational version(s) of executable software code shall be provided by the contractor to allow PMRB user feedback to be incorporated into final deliverable executable software code; e) interim version(s) of software documents (i.e., design manual, user manual, test/verification manual, metadata document, etc.) shall be provided by the contractor to allow PMRB user feedback to be incorporated into final deliverable software document versions; 17

19. Software QA Guidance Document • Software Quality Objectives f) executable software code shall be developed by the contractor in a manner which allows PMRB users to perform their required operational tasks through easy to use graphical user interface (GUI) controls as opposed to command line interface (CLI) control of executable software code (The contractor shall justify use of CLI-controlled software executable code where applicable); g) all executable software code developed by the contractor shall operate on all computer platforms (i.e., Windows XP, Linux 32-bit, Unix/Solaris/AIX, Linux 64-bit, etc.) as specified in the contract, grant, cooperative agreement, interagency agreement, CRADA, and/or technical direction of the EPA Project Officer/work assignment manager. 18

20. Software QA Guidance Document • Software Development The development process that the contractor implements to produce and deliver software source code and executable software code for PMRB research activities shall include, but not be limited to the elements explained in Section 3.1. If one or more elements in Section 3.1 are not required for a particular effort, the workplan, QAPP, or QA software plan shall clearly indicate why that element(s) can be omitted. Acceptance of any deviations from Section 3.1 is at the discretion of the EPA Project Officer/work assignment manager and the applicable EPA quality assurance manager. 19

21. Software QA Guidance Document Elements of Software Development Process The contractor shall implement a process to develop executable software code that includes the following principles of quality software development. The workplan, QAPP, or QA software plan shall clearly describe how each of the principles listed below will be implemented during the software development process: a) Creation of software design documents/operational specifications/data specifications b) Process to incorporate statement of work requirements into executable software code c) Reuse of current software source code (if available) as basis for further development 20

22. Software QA Guidance Document Elements of Software Development Process (cont’d) d) Generation of test plan(s)/scenario(s) to verify operation of executable software code e) Execution of test plan(s)/scenario(s) to verify operation of executable software code f) Development tools including test files, build scripts and code version control g) Methodology for development of GUI controls to facilitate system operations h) Process for estimating amount of software code (i.e. source lines of code - SLOC) i) Process for estimating cost/schedule/man-hours to develop executable software code 21

23. Software QA Guidance Document Elements of Software Development Process (cont’d) j)Process to determine tasks required to develop/test/deliver executable software code k) Criteria for determining successful completion of each software development phase l) Process to track/categorize/eliminate defects in delivered executable software code m) Process to determine selection of software language(s) to use for deliverables n) Process to determine what is included in each delivered software code version o) Process to manually examine source code listings for errors before delivery 22

24. Software QA Guidance Document Elements of Software Development Process (cont’d) p) Process to determine extent of use of third-party software, including assessment of benefits and limitations imposed by its use, and relevant aspects of licensing agreements The contractor shall ensure that all executable software code (including scripts, macros, etc.) developed for and delivered to PMRB in support of PMRB’s scientific research activities shall operate on all computer platforms (i.e., Windows XP, Linux 32-bit, Unix/Solaris/AIX, Linux 64-bit, etc.) as specified in the contract, grant, cooperative agreement, interagency agreement, CRADA, and/or technical direction of the EPA Project Officer/work assignment manager. 23

25. Software QA Guidance Document Elements of Software Development Process (cont’d) The contractor shall ensure that all executable software code developed for PMRB is tested on the same computer platform(s) that it is specified to operate on under EPA control. In other words, the contractor is prohibited from developing and testing executable software for computer platform A, when EPA specifies that the executable software must operate on computer platform B. The contractor shall solicit technical direction from the EPA Project Officer/work assignment manager to ensure that the contractor’s development and test computer platform(s) identically match the computer platform(s) on which EPA will operate the executable source code. 24

26. Software QA Guidance Document Source Code Format The contractor shall provide information in each module, function, subroutine, class, object, script, macro, etc., for all source code delivered to PMRB that describes the functionality of each major component. The categories of information to be provided in each source code file delivered to PMRB are shown in the example below: {EXCERPT}: EPA - EPA retains the unlimited rights to publish, translate, reproduce, deliver, distribute, and dispose of the technical data, computer software code or computer firmware contained herein. This technical data can not be copied, translated, reproduced, distributed or utilized in any way without the expressed written consent of EPA. This technical data may contain trade secrets and/or patented EPA processes. 25

27. Software QA Guidance Document Software Delivery Terms and Conditions The contractor shall follow the technical direction of the EPA project officer/work assignment manager to ensure that all software products (i.e., source code, executable code, and associated documentation, etc.) comply with the provisions of this document. The contractor shall ensure that all software products delivered to PMRB achieves the objectives of their respective scientific research project(s). As a minimum, the contractor shall provide to the EPA project officer/work assignment manager the following deliverables at the end of the period of performance of their project(s): a) a complete copy of the entire set of (uncompiled/uninterpreted) source code developed under their PMRB project(s); b) a complete set of executable software code; 26

28. Software QA Guidance Document Software Delivery Terms and Conditions c) a complete set of software documentation for all steps during the development process (i.e., design manual, user manual, test/verification manual, metadata document, etc.); d) a complete copy of any additional software and/or documentation resources (i.e., code scripts, macros, makefiles, assembly language routines, ‘cheat sheets’, etc.) used to develop the software source code and executable software code. The contractor shall solicit technical direction from the EPA Project Officer/work assignment manager (if not specified in the contract, grant, cooperative agreement, interagency agreement, CRADA, etc.) on the format(s) in which a), b), c), and d) are to be delivered (i.e., DVD, CD-ROM, via FTP Site/Transfer [for large files], etc.). 27

29. Software QA Guidance Document Software Delivery Terms and Conditions EPA shall retain unlimited rights to publish, translate, reproduce, deliver, distribute and dispose of the technical data, computer software code or computer firmware contained with the software code delivered by the contractor (within the constraints of any applicable third-party license agreements, previously approved, prior to software development by EPA). The technical data contained in the delivered software code, and the software code delivered to PMRB shall not be copied, translated, reproduced, distributed or utilized by the contractor in any way outside of PMRB research project activities without the expressed written consent of the EPA officer/work assignment manager. 28

30. Software QA Guidance Document • This document is being applied in two separate (current) cases to real EPA work assignments • Two different work assignments where models are being developed [names changed ] • Statistical Model (1) – Contractor Y • Model analyzes EPA-generated data • EPA wants to know correlation between • Measurement System/Technique “P” • Measurement System/Technique “Q” • How to relate “P” measurements in “Q” terms • Model used for ‘number crunching’ only • This is a newly-developed (original) model 29

31. Software QA Guidance Document • This document is being applied in two separate (current) cases to real EPA work assignments • Two different work assignments where models are being developed [names changed ] • Statistical Model (2) – Contractor Z • This model • Analyzes EPA data from two different sources • Used to reconcile two sets of measurements • Model is designed to be utilized by researchers • ‘Old’ model: 1) fix ‘flaws’; 2) enhance operation 30

32. Software QA Guidance Document • To guarantee that the provisions of this document are properly applied in your situation: • Ensure that your SOW/T&C portion of your contract/agreement directs the ‘contractor’ to incorporate the entire document by referenceinto the workplan that they develop for your project • Ensures model development quality elements are included in their work processes • Project Officer/COTR has a ‘checklist’ to verify if ‘contractor’ is implementing software QA standards/process • Follow-up with technical direction/inquiries to keep ‘contractor’ focused on EPA’s objective(s) 31

33. Software QA Guidance Document • Example Text (from Work Assignment Z): • TASK L.Implement Work Assignment Quality Assurance Requirements in Workplan. The contractor shall incorporate the provisions of the PMRB Software Development QA Guidance Document by reference into the workplan of this work assignment. The contractor shall implement the required software quality assurance guidance in the PMRB Software Development QA Guidance Document (as incorporated into the workplan) into the activities of this work assignment during the period of performance of this work assignment. 32

34. Software QA Guidance Document • Example Text (from Work Assignment Z): • (F) QA/QC Requirements for WA:The contractor shall incorporate the provisions of the PMRB Software Development QA Guidance Document in its entirety by reference into the workplan of this work assignment. The contractor shall prepare a Category IV (Cat IV) Software Development QAPP for this work assignment in conjunction with the workplan. The contractor shall incorporate the provisions of the PMRB Software Development QA Guidance Document in its entirety by reference into the Software Development QAPP of this work assignment. The contractor shall deliver the Software Development QAPP for this work assignment concurrently with delivery of the workplan for this work assignment (i.e., no later than 20 days after award of this work assignment). • SQA provisions incorporated into: workplan, QAPP 33

35. Software QA Guidance Document • Question (current status): Is this effort worthwhile? • Statistical Model (1) • Being designed as specified by EPA • EPA fully manages development process • Development cost is under budget • Development schedule is on target • Statistical Model (2) • Model quality has improved dramatically • Development is ahead of schedule • Contractor corrects errors before EPA reviews • Development cost is under budget • Only two data points thus far, but answer is yes! 34

36. Summary • EPA now has a 6-page software quality document • Designed as a resource for (non-software experts): • Contracting Officers (CO) • Contracting Officer Technical Representatives (COTR) aka WAM or WACOR • Task Order Project Officers (TOPO) • Delivery Order Project Officers (DOPO) • Project Officers (contracts, cooperative agreements, interagency agreements, CRADAs) • Organization QA Representatives • QA Managers/Directors • Anyone in EPA requiring ‘custom-made’ software

37. Summary • Used on two projects (contractor-developed software) • Software operation & documents greatly improved • EPA receives all required deliverables (no ‘gaffes’) • Provides ‘checklist’ of elements for quality software • Can be incorporated in statement of work/terms and conditions to give greater control over quality of deliverables (software and associated documentation) • Provides definitive guidance to contractors, etc., on what must be delivered to EPA and specifies EPA rights/control over delivered software, documentation, etc. • Augments EPA QA/G-5M by providing more detail • Standardizes how EPA obtains ‘custom’ software 36

38. Disclaimer • Disclaimer • Although this work was reviewed by EPA and approved for publication, it may not necessarily reflect official Agency policy.