Models in Environmental Regulatory Decision-Making

1. Models in Environmental Regulatory Decision-Making Report of the Committee on Models in the Regulatory Decision Process Presented at RASS- ISES Conference Call Chris Whipple, Committee Chair June 10, 2009

2. Origin of Study • Concerns over regulatory & judicial challenges to models • Against a backdrop of • Expanding reliance on models • Increasing model innovations • Increasing regulatory requirements(Data Quality Act and others)

3. Origin of Study • Paul Gilman appointed EPA Science Advisor • Revitalized the Council for Regulatory Environmental Modeling (CREM) • Moved CREM’s focus to model users • CREM tasked to • Develop guidance document, web-based knowledge base, regional workshops, stakeholder outreach • Engage the National Academy of Sciences

4. NRC Task Statement A National Research Council committee will provide advice concerning the development of guidelines and a vision for the selection and use of models at EPA. The committee will consider cross-discipline issues related to model use, performance evaluation, peer review, uncertainty, and quality assurance/quality control.The objective …. [is] to provide a report that will serve as a fundamental guide for the selection and use of models in the regulatory process. As part of its work, the committee will need to carefully consider the realities of EPA's regulatory mission so as to avoid an overly prescriptive and stringent set of guidelines.

5. Committee Members • Chris G. Whipple, Environ, Inc. (Chair) • M. Bruce Beck, University of Georgia • Clayton Clark, University of Florida • Robert T. Clemen, Duke University • Judith A. Graham, American Chemistry Council • Louis J. Gross, University of Tennessee • Winston Harrington, Resources for the Future • Philip Howard, Syracuse Research Corporation • Kimberly L. Jones, Howard University • Thomas E. McKone, University of California • Naomi Oreskes, University of California • Spyros N. Pandis, Carnegie Mellon University • Louise M. Ryan, Harvard School of Public Health • Michael L. Stein, The University of Chicago • Wendy E. Wagner, University of Texas School of Law

6. Fundamental Definition • What is an Environmental Regulatory Model? • “A computational model used to inform the environmental regulatory process. Some models are independent of a specific regulation …. Other models are created to provide a regulation-specific set of analyses ….. The approaches can range from single parameter linear relationship models to models with thousands of separate components and many billions of calculations.” • Implications of this definition • Wide array of models under Committee’s consideration • Wide array of regulatory applications

7. Information Gathering • Workshop #1 – breadth of environmental regulatory modeling at EPA and issues associated with these activities • Role of EPA regional offices • Interactions of EPA, states, and consultants • Variability in scale of model applications • Environmental and business group perspectives • Workshop #2 – focus on 3 specific issues in-depth • Proprietary models – motivations, proprietary features, mitigation • Peer review – types and elements of peer review • Uncertainty analysis – methods and use in decisions

8. Information Gathering • Workshop #3 - new sources of information, future health risk assessment modeling, and the role of models in decision-making • Genomics to environmental satellites • Potential to mechanistically model human responses • Decisions made by non-modelers

9. Study Approach Committee considered broad patterns of models usage in making recommendations EPA has significantly advanced the science of environmental modeling—report intended to help provide an cross-agency vision and principles for the use of models in the future Never say “never” or “always”

10. Model Use in Regulatory Processes Models always constrained by computational limitations, assumptions, and knowledge gaps Environmental models can never be completely “validated” in the traditional sense. But they can be “evaluated” Regulatory models are typically used to describe important, complex, and poorly characterized problems Models in the regulatory process best seen as tools providing inputs, as opposed to “truth generating machines”

11. Implications for Regulatory Model Use • Evaluation of regulatory models requires different tradeoffs than those for research models • Not simply how well do model estimates match observations, but also how reproducible, transparent, and useful a model is to the regulatory decision • Requires regulatory models be managed to be updated in a timely manner and assist users and others to understand conceptual basis, assumptions, input data requirements, and life history

12. List and Structure of Recommendations • MODEL EVALUATION • Life Cycle Model Evaluation • Peer Review • Assessing and Communicating Uncertainty • The Interdependence of Models and Measurements • Retrospective Analysis of Models* • PRINICIPLES FOR MODEL DEVELOPMENT, SELECTION, AND APPLICATION • Model Parsimony* • Extrapolation* • Proprietary Models • MODEL MANAGEMENT • Models and Rulemakings • Model Origin and History* • Improving Model Accessibility*

13. Evaluation of regulatory models is the process of deciding whether and when a model and its application is appropriate Evaluation is a multifaceted activity—peer review, corroboration of results with data and other information, QA/QC checks, uncertainty and sensitivity analyses, and other activities Not a one time event—evaluation of regulatory models should continue throughout regulatory applications and revisions to the model Life-Cycle Model Evaluation

14. Life-Cycle of a Model Problem Formulation Conceptual Model Constructed Model Model Use

15. Life-Cycle Model Evaluation • All models should have a life-cycle evaluation plan of a size and complexity commensurate with its regulatory significance • Plan should address how model evaluation will occur throughout a model’s life cycle • The committee did not make organizational recommendations of how EPA should achieve this • A conceptual commitment to life cycle model evaluation is needed

16. Peer Review Peer review activities are an integral component of the evaluation process Some simple, uncontroversial models might require little or no peer review But many models require detailed and multiple peer reviews that involve time and resources more extensive than a report or journal paper—review of only model documentation is not sufficient for important models Some regulatory model peer reviews will involve reviewing the model code and its documentation, and comparing the model results against known test cases

17. Assessing/Communicating Uncertainty • Two Approaches • Represent uncertainties probabilistically and calculate the probability distribution of any model result • difficult to carry out • obscures the sensitivities of the outcome to individual sources of uncertainty • Scenario assessment and/or sensitivity analysis • often more transparent • may ignore important information corresponding to other scenarios not included in assessment and whatever is known about their relative likelihoods

18. Assessing/Communicating Uncertainty Many instances require probabilistic methods to properly characterize uncertainties, propagate them through the modeling exercise, and clearly communicate the overall uncertainties Recommend the use of case-specific hybrid approaches in which some unknown quantities are treated probabilistically, and others can be manipulated in a scenario-assessment mode by the decision makers Requires communication between modelers and decision-makers

19. Models and Data Interdependence of measurements and modeling must be considered during development of a conceptual model Requires development of adaptive strategies to advance data collection and model development Future will see vast new sources of information- to maximize the effectiveness of new data collection efforts, models must be able to use this information AND guide the design of data collection

20. Proprietary Models Use of proprietary models can produce distrust among regulated parties and other groups Recommends that EPA adopt a strong preference for nonproprietary software Only use proprietary models when a clear case can be made for their advantages Proprietary models should be subject to same evaluation requirements as for public models

21. Models and Rulemaking EPA may perceive that rigorous life cycle evaluation of models and documenting such features in need of improvement—may expose models to greater risk of challenges and trigger lengthy regulatory process Life cycle evaluation practices may be much easier if such activities were considered to satisfy regulatory requirements, such as those of the Information Quality Act EPA could consider establishing a generic rule for the process of evaluation and adjustment of models used in rule-making to provide adequate opportunities for public comment and revision of an individual model without triggering the need for a separate rule-making for each model revision

22. List and Structure of Recommendations MODEL EVALUATION Life Cycle Model Evaluation Peer Review Assessing and Communicating Uncertainty The Interdependence of Models and Measurements Retrospective Analysis of Models* PRINICIPLES FOR MODEL DEVELOPMENT, SELECTION, AND APPLICATION Model Parsimony* Extrapolation* Proprietary Models MODEL MANAGEMENT Models and Rulemakings Model Origin and History* Improving Model Accessibility*