Annual Report by Work Group 2 on Uncertainty Analysis and Parameter Estimation

1. Annual Report by Work Group 2 on Uncertainty Analysis and Parameter Estimation Tom Nicholson, U.S. Nuclear Regulatory Commission Mary Hill, U.S. Geological Survey 2014 ISCMEM Annual Public Meeting February 26, 2014 DOE Headquarters, Washington, DC

2. Outline • Work Group 2 (WG2) Objective and Goals • Members and Participants • Activities and Technical Projects • Seminars at the WG2 Meetings • Methodologies, Tools and Applications • Forward Strategy • Recommendations for FY2014

3. Work Group Objective • Coordinate ongoing and new research conducted by U.S. Federal agencies on: parameter estimation uncertainty assessment in support of environmental modeling and applications • Focus on strategies and techniques • Includes sensitivity analysis What is needed to achieve this objective? Coordination of research staff and their management thru efficient and targeted use of our limited resources.

4. Electrical Conductivity Butler et al. Work Group Goals Batch scale (0.01m) • Basics: • Develop a creative, collaborative environment to advance • parameter estimation in the context of model development . • sources of uncertainty in the context of model predictions. • Develop a common terminology. • Identify innovative applications. • Existing Tools: Identify, evaluate, and compare available analysis strategies, tools and software. • New Tools: Develop, test, and apply new theories and methodologies. • Exchange: Facilitate exchange of techniques and ideas thru teleconferences, technical workshops, professional meetings, interaction with other WGs and ISCMEM Web site links: https://iemhub.org/groups/iscmem • Communicate: Develop ways to better communicate uncertainty to decision makers (e.g., evaluation measures, visualization). Column scale (0.1 m) Intermediate scale (2m) Tracer test scale (1-3m) Geophysics (2-200m) Plume scale (2000m)

5. Tom Nicholson, NRC, co-Chair Mary Hill, USGS, co-Chair Ming Zhu, DOE Tommy Bohrmann, EPA Gary Curtis, USGS Bruce Hamilton, NSF Yakov Pachepsky, USDA-ARS Tom Purucker, EPA-Athens Yoram Rubin, UC Berkeley Brian Skahill, USACOE Matt Tonkin, SSPA Gene Whelan, EPA-Athens Steve Yabusaki, PNNL Ming Ye, Florida State U Sanja Perica, NOAA/NWS Larry Deschaine, HydroGeologic, Inc. Boris Faybishenko, LBNL Pierre Glynn, USGS Philip Meyer, PNNL Debra Reinhart, NSF You? Members and Participantsfrom U.S. Federal agencies,universities, and industry

6. Activities: Seminars We conduct seminarsto: • review and discuss ongoing research studies and software development • formulate proposals for field applications How to retrieve Precipitation Frequency estimates with confidence limits? from 6/26/2013 seminar by Sanja Perica, NOAA/NWS on NOAA Atlas 14

7. Seminar at WG2 in FY2013 • NOAA Atlas Precipitation – Frequency Atlas of the U.S.: Products and Methods by Sanja Perica, Director, Hydrometeorological Design Studies Center, Office of Hydrology, National Weather Service, NOAA. Current NOAA/NWS Precipitation Frequency (PF) documents by State location: http://www.nws.noaa.gov/oh/hdsc/currentpf.htm Access NOAA/NWS’s Precipitation Frequency Data Server (PFDS) at: http://hdsc.nws.noaa.gov/hdsc/pfds/index.html

8. Seminars at WG2 Teleconferences(continued) NOAA Atlas 14 • Simulation used to construct 90% confidence intervals (i.e., 5% and 95% confidence limits) at stations. • Algorithm adjusted to account for inter-station correlation. • Estimates interpolated on 30 arc-sec grid. • Uncertainty analysis and sensitivity analyses are included in the simulation result analyses.

9. WG2 ActivitiesUCODE_2014 UCODE - A Computer Code for Universal Inverse Modelinghttp://pubs.usgs.gov/tm/2006/tm6a11/pdf/TM6-A11.pdf Eileen P. Poeter, Colorado School of Mines and the Integrated Groundwater Modeling Center, Emeritus Mary C. Hill, U.S. Geological Survey Dan Lu, Oak Ridge National Laboratory Steffen Mehl, California State University-Chico 9 9999

10. Selected New Features • Evaluate Uncertainty with Markov-Chain Monte Carlo (MCMC) • Use for non-Gaussian distributions. Often this means nonlinearity produces local minima • New stacked sensitivity graphs • Show parameter importance and contributions from different types of observations

11. Evaluate Uncertainty with Markov-Chain Monte Carlo (MCMC) • The DREAM algorithm used (Vrugt et al., 2009) is able to search very complicated distributions • This example is for an very nonlinear analytic function from Lu Ye Hill, 2012, WRR

12. Quantifying Uncertainty: Tests using a groundwater model of a synthetic system • Lu, Ye, Hill, 2012, WRR • Investigate methods that take 10s to 100,000s of model runs (Linear to MCMC) • New version of UCODE with MCMC

13. Test Case True • MODFLOW • 3.8 km x 6 km x 80m • Top: free-surface • No-flow sides, bottom • Obs: 54 heads, 2 flows, lake level, lake budget • Parameters: 5 for • recharge • KRB • confining unit KV • net lake recharge, • vertical anisotropy K : 1 (HO), 3 (3Z), or 21 (INT) Data on K at yellow dots Int 3Z

14. Predictions, Confidence Intervals Model runs 100 1,600 480,000 True HO 3Z INT Name of alternative model…….. Statistics to use when the truth is unknown Lesson about uncertainty • May be more important to consider the uncertainty from many models than use a computationally demanding way to calculate uncertainty for one model

15. New stacked sensitivity graphs • Here, parameter T16 is most important, P110 is least. (T is a transmissivity, P is porosity) • Head obs () contribute to the T parameters. • Time of travel (), concentration(), and source () (fraction of water from different locations) are broadly important Hanson, Kauffman, Hill, Dickinson, Mehl, 2013, MODPATH-OBS documentation, USGS

16. The following slides show the problem for which the stacked graph was created

17. Parameters Observations • 5 K • 5 Porosities 15 Heads • 5 Proximity to expected location by a defined time • 14 Time of travel • 50 “Concentrations” • 25 Source water type Particle path figures are provided to show simulation tests

18. Well 3 Trench Particle paths Pit Well 2 Farm Obs well 1 Well 3 Trench Pit Well 2 Farm Obs well 1

19. Starting parameter values Estimated parameter values PCE concentrations simulated to reach well 2 from the trench and the pit

20. Local grid refinement (LGR) to simulate pumping. Particles transported through the locally refined grid. Parameters defined in zones: K, porosity. Constant Head=11 Constant Head=0

22. Sensitivity analysisNew DELSA. Compare to global SA methodsUSGS-NCAR collaborative study with Mary Hill, USGS; Martyn Clark, NCAR; Olda Rakovec, U of Wageningen, and others from his university. • Rakovec + 2014 WRR • New DELSA multi-scale statistic (1,000 model runs) • DELSA: Distributed Evaluation of Local Sensitivity Analysis • Compare to global variance-based Sobol’ statistics (10,000 model runs) • Rainfall-runoff problem • FUSE modular model (Clark et al. 2008, WRR)

23. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR IZONE LAYR1 SOILR LAYR2 GWATR GFLWR FUSE supports quantifying uncertainty using multiple models… PRMS SACRAMENTO TOPMODEL ARNO/VIC

24. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR IZONE LAYR1 SOILR LAYR2 GWATR GFLWR Define model building decisions—1 PRMS SACRAMENTO UPPER LAYER ARCHITECTURE TOPMODEL ARNO/VIC

25. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR IZONE LAYR1 SOILR LAYR2 GWATR GFLWR Define model building decisions—2 PRMS SACRAMENTO LOWER LAYER ARCHITECTURE / BASEFLOW TOPMODEL ARNO/VIC

26. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR IZONE LAYR1 SOILR LAYR2 GWATR GFLWR Define model building decisions—3 PRMS SACRAMENTO PERCOLATION TOPMODEL ARNO/VIC

27. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR IZONE LAYR1 SOILR LAYR2 GWATR GFLWR Define model building decisions—4 PRMS SACRAMENTO SURFACE RUNOFF TOPMODEL ARNO/VIC

28. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR + IZONE + + LAYR1 + + SOILR + LAYR2 GWATR GFLWR and mix architecture + parameterizations… PRMS SACRAMENTO TO CREATE HUNDREDS OF MODELS, ALL WITH DIFFERENT STRUCTURE TOPMODEL ARNO/VIC

29. Field problemBelgian watershedConsider RMSE

30. Parameters common to all 5 alternative models constructed using FUSE Global method produces averages. DELSA provides a distribution DELSA first-order sensitivities [-]

31. Dig deeper into DELSA for FUSE-016 DELSA Sobol DELSA first-order sensitivity

32. Potential reason for variable sensitivities from Kavetski and Kuczera 2007 WRR

33. PCTIM IMPZR ADIMC ADIMP RZONE IFLWR UZTWC UZFWC LZONE LZFSC LZFPC LZTWC GFLWR IZONE LAYR1 SOILR LAYR2 GWATR GFLWR FUSE supports quantifying uncertainty using multiple models… PRMS SACRAMENTO TOPMODEL ARNO/VIC

34. Lesson from DELSA • Utility of detailed SA, including evaluation at many sets of parameter values and many models. • Also, over time.

35. Forward Strategy Energize the science and technology thru closer linkage to decision making: • better understand the methods being used in parameter estimation and uncertainty analyses • establish a base set of model sensitivity analysis and uncertainty evaluation measures, in addition to the other performance measures • use and compare different methods in practical situations

36. Recommendations for FY2014 • Assist development and creation of other working groups • Take advantage of the relevance of uncertainty and parameter estimation to all environmental modeling and monitoring fields. • Develop and conduct joint ISCMEM teleconferences • WG1 (Software System Design; design of uncertainty and parameter estimation software and data fusion) • WG3 (Reactive Transport Models and Monitoring; support decision making) • Act as an incubator to build support for new ideas • Proposed WG on monitoring based on the importance of monitoring to uncertainty and parameter estimation, and visa versa • Sponsor technical workshops on endorsed studies • ISCMEM Website • Use EPA’s iemHUB to enhance Information Transfer of Technical Reports and Data Sources

37. https://iemhub.org/groups/iscmem