Streamlining Uncertainty Conceptual Model and Scenario Uncertainty

1. Streamlining UncertaintyConceptual Model and Scenario Uncertainty FRAMES-2.0 Workshop U.S. Nuclear Regulatory Commission Bethesda, Maryland November 15-16, 2007 Pacific Northwest National Laboratory Richland, Washington

2. Model Applications • Regulatory and design applications of hydrologic models of flow and contaminant transport often involve using the models to make predictions of future system behavior • Performance assessment of new facilities (safety evaluation, environmental impact assessment) • Monitoring network design for contaminant detection or performance monitoring • License termination • Design of a subsurface contaminant remediation system 2

3. New Reactor Potential Model Applications • Assessing effects of accidental releases on ground and surface waters • groundwater flow pathways • transport characteristics • Assessing flood design bases • Stream flooding • Local flooding, site drainage • Impacts of water use • Watershed analysis – impacts on other users of water source upstream and downstream, particularly during drought conditions 3

4. Framework for the Application of Hydrologic Models to Regulatory Decision Making • History Matching - reproduce observed behavior • Demonstrate understanding of site behavior • Provide confidence in use of models to support decisions • Prediction – forecast future behavior • Apply model results to decisions • For risk-informed decision making, provide estimates of risk Model Application for Comparison with Regulatory or Design Criteria Model Development & Evaluation 4

5. Model Predictive Uncertainty Quantifies Element of Risk • In general, uncertainty is assessed in the history-matching period and propagated into the predictive period • Reduce these uncertainties by collecting additional data • Some uncertainties only apply in the predictive period • Irreducible characteristics of the system being modeled 5

6. Prediction Uncertainty • Model conceptualization uncertainty • A hypothesis about the behavior of the system being modeled and the relationships between the components of the system • Each site is unique and heterogeneous/variable. Behavior typically involves complex processes. Site characterization data is limited. • Assessed in history-matching period, applied in the predictive period • Parameter uncertainty • Model-specific quantities required to obtain a solution • Measurement/sampling errors. Disparity among sampling, simulation, and actual scales of the system. • Assessed in history-matching period, applied in the predictive period • Scenario uncertainty • Future state or condition that affects the hydrology • Historical record not representative of future conditions – process variability, limited historical record, land/water use changes, climate change • Applies to predictive period only 6

7. Model Uncertainty • Common to rely on a single conceptual model of a system. This approach is inadequate when there are: • different interpretations of data • insufficient data to resolve differences between conceptualizations 7

8. Failure to Consider Model Uncertainty • Has two potential pitfalls: • rejection by omission of valid alternatives (underestimates uncertainty) • reliance on an invalid model (produces biased results) 8

9. 9

10. How to Proceed? • Desirable characteristics of a methodology for uncertainty assessment • Comprehensive: as many types of uncertainty as possible should be included • Quantitative: it should be possible to compare results with regulatory criteria or design requirements • Systematic: able to be applied to a wide range of sites and objectives and to enable the common application of computer codes and methods 10

11. Deterministic Approach • Assumptions • Model parameters are correct • Model is correct • Scenario is known 11

12. Parameter Sensitivity Approach • Assumptions • Model parameters are unknown • Model is correct • Scenario is known 12

13. Parameter Sensitivity Approach • Results • Probability of peak dose represents the degree of plausibility of the model result • ? indicates that the actual values of the probabilities are unknown; statements about the relative values may be possible • Bounding (conservative) analysis: the desired predicted value represents the worst plausible behavior of the system • Limitations • Can’t quantitatively estimate risk since probabilities are unknown [risk = p(D > 25 mrem/yr)] • Significance of bounding case must be assessed to avoid over-conservatism • Significant sources of uncertainty not included 13

14. Parameter Uncertainty Approach • Assumptions • Model parameters are uncertain • Model is correct • Scenario is known 14

15. Parameter Uncertainty Approach • Method • Assign joint probability distribution to model parameters and propagate through the model (e.g., using Monte Carlo simulation) • Results • Peak dose probability density represents the degree of plausibility of the model result • Quantitative estimates of probabilities can be computed • Quantitative estimates of risk can be computed • Limitations • Joint probability distribution of parameters must be determined • May be computationally expensive • Significant sources of uncertainty not included 15

16. Conceptual Model Sensitivity Approach 16

17. Conceptual Model Sensitivity Approach • Method • Postulate alternative conceptual models for a site that are each consistent with site characterization data and observed system behavior, • Results • Each model is used to simulate the desired predicted quantity • Parameters of each model (which may be different) are represented using a joint probability distribution • Limitations • Without a quantitative measure of the degree of plausibility of model alternatives, it is impossible to determine the risk of a decision based on the model predictions • A conservative approach to model uncertainty relies on an implied belief that the most conservative model has a non-negligible degree of plausibility • Requires formulation & simulation of multiple models 17

18. Quantitative Model Uncertainty • Assign a discrete probability distribution to the conceptual model alternatives • Analogous to the interpretation of parameter probability, the discrete model probability distribution represents the degree of plausibility of the model alternatives • What quantity to compare with regulatory/design criteria? 18

19. Probability-Based Model Selection • Use the model with the highest probability for predictions • Potentially biased result if significant probability with alternative models • If variance due to model uncertainty is desired, must compute predicted value using each model 19

20. Conservative Model Selection • Use the model with the most significant consequence • How little probability must lie with the highest consequence model before it is judged implausible? • Consequence must be computed with each model to determine the conservative model 20

21. Probability-Weighted Model Averaging 21

22. Probability-Weighted Model Averaging • Method • Model predictions are combined using a weighted average with the weight for each model’s prediction consisting of that model’s probability • Results • Model-averaged probability density function represents the degree of plausibility of the predicted value that takes into consideration the joint effect of parameter and model uncertainties • Reduces bias • Less likely to underestimate predictive uncertainty • Consistent treatment of parameter and model uncertainties • Quantitative estimates of risk can be computed from the model-averaged result 22

23. Probability-Weighted Model Averaging • Limitations • Model probability is a relative measure with respect to the other model alternatives considered • Requires specifying model probability distribution • Requires formulating & simulating multiple models • Doesn’t consider scenario uncertainty 23

24. Model-Averaging Informative Results • Results suggest collection of additional data to better discriminate between models (i.e., to modify model probabilities until one model dominates) • Exceedance probability and 90th percentile suggest that a conservative regulatory action may be preferred • based on a fully-informed consideration of model and parameter uncertainty (i.e., risk), rather than on adoption of the most conservative model 24

25. Scenario Uncertainty: Unknown Future State or Condition of the System • Scenario uncertainty can’t be reduced through the application of data (unlike parameter & model uncertainty) 25

26. Scenario Uncertainty Sensitivity Approach 26

27. Scenario Averaging Approach 27

28. Probability-Weighted Scenario Averaging • Method • Model-averaged predictions for each scenario are combined using a weighted average with the weight for each scenario’s prediction consisting of that scenario’s probability • Results • Scenario- and model-averaged probability density function represents the degree of plausibility of the predicted value that takes into consideration the joint effect of parameter, model, and scenario uncertainties • Quantitative estimates of risk can be computed from the scenario- and model-averaged result • Limitations • Requires specifying scenario probabilities • Requires simulations of each model under each scenario 28

29. Scenario-Averaging Informative Results • Mean dose results straddle regulatory threshold suggesting that a conservative regulatory action may be preferred • based on a fully-informed consideration of model, parameter, and scenario uncertainty (i.e., risk), rather than on adoption of the most conservative modeling choices 29

30. NRC Staff Application of Probability-Weighted Model Averaging MODEL 2 MODEL 3 MODEL 4 MODEL 5 30

31. Alternative Model Development • Models developed using Groundwater Modeling System (GMS). • Model 2: average values for hydraulic conductivity, recharge, and evapotranspiration • Model 3: average values for hydraulic conductivity and evapotranspiration, zonal values for recharge • Model 4: average value for hydraulic conductivity, zonal values for recharge and evapotranspiration • Model 5: same as model 4 with a general head boundary, recharge, and evapotranspiration 31

32. Model & Scenario Averaging Application Simulation Results under Two Scenarios (Well 399-1-1) 32

33. Scenario Average (Baseline 70%) 33

34. Project Objectives • Improve access to the uncertainty assessment methodology by integrating methods with FRAMES • Provide guidance on the use of model abstraction techniques to generate plausible and realistic alternative conceptual models for a site • Parameter estimation • Quantitative model comparison • Simulation using multiple models and scenarios • Demonstrate using a realistic application relevant to NRC/NRO analyses 34

35. Project Schedule • Summer 2008 • Implementation of methods completed • NRC workshop • Summer 2009 • Completion of application • NRC workshop 35